How Transformers Work, Part 1: Tokenizers

Following the Transformer, Part 1: Tokenizers

At first glance, a tokenizer looks like nothing more than a mundane preprocessing step — a tool that splits sentences into pieces and converts them to numbers. Most explanations stop there. But once you look at LLMs more carefully, it becomes clear that the tokenizer is not some peripheral accessory bolted onto the outside of the model. It is the mechanism that determines the fundamental unit by which the model reads the world.

Humans read sentences as words and meaning. Models read arrays of token IDs. Whether 안녕하세요 is a single token, split into 안녕 and 하세요, or broken down further into byte fragments — this changes the shape of the input the model will see from start to finish. The same Korean sentence can have very different lengths depending on the tokenizer, and different lengths mean different attention costs. Long-document handling, Korean-language performance, code generation, and special token design are all affected by this choice.

The tokenizer converts a raw string into an array of token IDs that the model can compute over.

The model does not read characters directly

You cannot feed raw text directly into a Transformer. Neural networks do not operate on strings — they operate on numeric tensors. So text must first be split into tokens, and each token must be mapped to an integer ID from the vocabulary.

Say a tokenizer splits a sentence like this:

입력 문장: 나는 토크나이저를 공부한다
토큰:     [나는, 토크, 나이저, 를, 공부, 한다]
토큰 ID:  [8123, 441, 9201, 317, 2304, 901]

What the model actually receives is not the string 나는 토크나이저를 공부한다 but an integer array like [8123, 441, 9201, 317, 2304, 901]. From there, the embedding layer maps each ID to a vector, and the Transformer blocks compute relationships among those vectors.

A common misconception is that tokens equal words. In English, unbelievable might be split into un, believ, and able. In Korean, 공부했습니다 might become 공부, 했, and 습니다. In code, units like def, Ġfunction, and () become tokens. Whitespace matters too. In GPT-family tokenizers, hello and hello are frequently represented as distinct tokens.

A tokenizer is a bidirectional contract between strings and token IDs:

encode: "Transformer" -> [41762]
decode: [41762] -> "Transformer"

If the tokenizer used during training differs from the one used at inference, the model loses the familiar input distribution it was trained on. Having just the model parameters is not enough. The vocabulary, merge rules, normalizer, and special token configuration all have to match — together, they define the model.

Why not just feed in individual characters?

The simplest approach is to tokenize one character at a time:

나는 밥을 먹었다
-> [나, 는,  , 밥, 을,  , 먹, 었, 다]

This keeps the vocabulary small. Even covering Korean, English, digits, and symbols, the vocabulary stays far smaller than any word-level dictionary. It also handles unseen words cleanly. Sounds great.

The problem is length. Characters are so fine-grained that the number of tokens needed to represent the same sentence balloons. Transformer self-attention is sensitive to sequence length — doubling the token count roughly quadruples the number of attention pairs. In practice, FlashAttention, KV caching, and chunking mitigate this considerably, but the fundamental fact remains: the number of input tokens drives both cost and latency.

Word-level tokenization has the opposite problem:

나는 밥을 먹었다
-> [나는, 밥을, 먹었다]

Length gets shorter, but any out-of-vocabulary word becomes a problem. This is why [UNK] tokens appeared so often in older NLP models. There is no practical way to pre-populate a word dictionary with strings like ChatGPT, Qwen3, hrletsgo.me, or calculateUserEmbedding().

Today's mainstream LLM tokenizers sit between these two extremes. Frequent substrings are grouped into larger chunks; rare strings are broken into smaller pieces. English words end up as word-like fragments, Korean postpositions and verb endings become smaller pieces, and URLs and code can fall all the way down to the byte level.

Character-level tokenization produces long sequences; word-level tokenization struggles with unseen words. Subword tokenization is the compromise between the two.

BPE: merge the fragments that appear together most often

BPE stands for Byte Pair Encoding. It was originally proposed as a compression algorithm and was later adapted for subword tokenization in NLP. The landmark NLP paper applying BPE is Sennrich et al., Neural Machine Translation of Rare Words with Subword Units.

The idea is simple: find the two fragments that appear adjacent most frequently and merge them into one. Repeat until the vocabulary reaches the target size.

A small example. Suppose the training corpus contains these words:

low lower lowest

Start by splitting into individual characters:

l o w
l o w e r
l o w e s t

The most frequent adjacent pair is l o. Merge it into lo:

lo w
lo w e r
lo w e s t

Next, if lo w is the most frequent pair, merge it into low:

low
low e r
low e s t

Continuing this process, pieces like low, er, and est end up in the vocabulary. The model does not need to memorize lowest as a whole word — it can represent it as the combination of low and est. This is how BPE handles rare words without discarding them as [UNK].

In practice, BPE training counts pair frequencies over a much larger corpus to build a merge table. At inference time, that same merge table is applied to split any string by the same rules. The more often two substrings co-occur in the training data, the more likely they are to be merged into a single token.

학습 결과 merge rule 일부가 다음과 같다고 가정한다.

("t", "h") -> "th"
("th", "e") -> "the"
("i", "n") -> "in"
("ing", "</w>") -> "ing</w>"

These merge rules are more powerful than a simple word list, because they let the tokenizer assemble substrings it has never seen as complete units, by applying the rules bottom-up.

BPE usually comes with a pre-tokenization step

One part that often gets glossed over in BPE explanations is pre-tokenization. Traditional BPE-based NLP tokenizers typically split text first — using whitespace, punctuation, or a language-specific morphological analyzer — and then apply BPE within each resulting piece.

For English, whitespace-based splitting works reasonably well:

I love tokenizers.
-> [I, love, tokenizers, .]

For Korean, whitespace alone is insufficient. Endings and postpositions attach directly to stems — 먹었습니다, 먹었고, 먹었지만 — and spacing quality varies wildly across datasets. This is why Korean tokenizer designs often use a morphological analyzer like MeCab as a pre-tokenizer.

The EXAONE 3.0 technical report describes applying byte-level BPE after MeCab-based pre-tokenization to handle both Korean and English. Expressed roughly as a pipeline:

원문
-> MeCab 등으로 형태소/어절 단위 pre-tokenization
-> 각 조각에 byte-level BPE 적용
-> token ID 배열

This design makes sense for Korean. The morphological analyzer captures the internal structure of Korean eojeols to a reasonable degree, and BBPE handles rare expressions and symbols down to the byte level.

There are trade-offs. A pre-tokenizer encodes language-specific rules. When data mixes English, Korean, code, math, emoji, and URLs, the outcome depends heavily on which rules fire first. From the model's perspective, it continuously learns the token distribution that those rules produce.

Byte-level encoding: never get an unknown character

Byte-level tokenizers drop below the Unicode character level and process text as raw UTF-8 bytes. Every text file is ultimately a byte array. A byte-based vocabulary can, in principle, represent any character — novel CJK characters, garbled symbols, new emoji, unusual control characters — all without ever needing [UNK].

"가"의 UTF-8 byte: EA B0 80
"A"의 UTF-8 byte: 41
"🙂"의 UTF-8 byte: F0 9F 99 82

Byte-level tokenization is robust. Web data is messy: URLs, HTML entities, code, multilingual text, and broken encodings all show up together. Working at the byte level at least eliminates the concept of an unrepresentable character.

The downside is that the tokens look messy to human eyes. The Ġ character that shows up in GPT-2-family tokenizers exists because whitespace is encoded in a non-standard way. That symbol is not a meaningful character — it is an internal marker used to preserve whitespace position so decoding can reconstruct the original string.

Byte-level BPE (BBPE) starts from individual bytes and learns BPE merges on top of them. Neural Machine Translation with Byte-Level Subwords showed that byte-level subwords reduce vocabulary coverage problems in multilingual and noisy-text settings.

BBPE largely eliminates the worry about out-of-vocabulary characters. The trade-off is that the same sentence may require more tokens, and CJK characters in particular can expand significantly when they fall through to the byte level. A well-designed tokenizer makes a good compromise here: group common Korean fragments into large chunks, and reserve byte-level representation only for genuinely rare characters.

SentencePiece: don't split on whitespace first

SentencePiece is Google's open-source subword tokenizer library. It implements both BPE and the Unigram Language Model tokenizer. Its widespread adoption stems from its ability to train without relying heavily on whitespace as a word boundary signal.

In English, whitespace is a fairly reliable word delimiter. In Korean, Japanese, and Chinese, whitespace-based splitting is much weaker. Japanese and Chinese use little or no spacing; Korean has spacing, but it does not align consistently with morpheme boundaries.

SentencePiece treats the input as raw text and trains directly on it. Whitespace is treated as just another character, and its position is typically marked with the ▁ symbol.

원문: 나는 토크나이저를 공부한다
SentencePiece식 표기 예:
▁나는 ▁토크 나이저 를 ▁공부 한다

Here ▁ means "there was a space before this token." This matters for decoding — if whitespace is simply consumed as a split delimiter, you cannot faithfully reconstruct the original string from the token array alone.

The Unigram LM tokenizer that SentencePiece implements has a different training philosophy from BPE. BPE incrementally merges frequent pairs. Unigram starts with a large set of candidate subword pieces, learns a probability model over segmentations that best explains the corpus, and then prunes away pieces that are not needed. Both produce subword tokenizers, but the criteria for which pieces survive differ.

BPE's behavior is governed by a deterministic sequence of merges. Unigram is oriented around selecting the highest-probability segmentation among many candidates. This makes Unigram-based tokenizers a natural fit for techniques like subword regularization: instead of always splitting a sentence the same way, you sample slightly different segmentations with some probability during training, which discourages the model from over-fitting to specific token boundaries.

BPE merges frequent pairs; Unigram learns a probability distribution over candidate pieces and selects which ones to keep.

GPT-2, ModernBERT, and EXAONE side by side

GPT-2 uses a byte-level BPE tokenizer. Its vocabulary size — 50,257 — is a number you encounter often when working with GPT-2 models and tokenizers together.

ModernBERT is a BERT-family model, yet it uses OLMo's BPE tokenizer rather than WordPiece. The ModernBERT paper explicitly states a vocabulary size of 50,368 and a special token set of ['[UNK]', '[SEP]', '[PAD]', '[CLS]', '[MASK]']. The era of assuming every BERT-family model uses WordPiece is over. Model architecture lineage and tokenizer choice need to be considered separately.

EXAONE 3.0 uses MeCab pre-tokenization combined with BBPE for Korean. This pairing is a deliberate attempt to capture Korean's morphological characteristics while retaining the robustness of byte-level encoding. Picking a tokenizer carelessly for a Korean LLM directly affects how much meaning fits within a given context length. A model that tokenizes a single Korean sentence into far more tokens than the equivalent English text is already at a disadvantage when processing long documents.

A quick comparison:

Special Tokens Are Control Characters for Communicating with the Model

A tokenizer vocabulary contains more than just ordinary text fragments. It includes special tokens like [PAD], [UNK], [CLS], [SEP], [MASK], <s>, </s>, <bos>, and <eos>. Chat models also add role tokens such as <|user|>, <|assistant|>, and <|system|>.

The model learns to treat these tokens as carrying specific meaning during training.

[CLS] position that aggregates the representation of the entire sentence
[SEP] separates two sentences or segments
[MASK] placeholder for masked positions in masked language modeling
[PAD] padding to equalize batch sequence lengths
<EOS> signals end of generation

In generative models, the EOS token is especially important. The model learns "how to end a sentence" as part of training. If EOS probabilities are handled incorrectly, responses get cut off too early or run on indefinitely. This is why accidentally dropping the EOS token from fine-tuning data is a critical mistake.

Chat templates are also tied to the tokenizer. The same conversation produces a different token sequence depending on which template is used.

<|system|>
너는 도움이 되는 assistant다.
<|user|>
토크나이저가 뭐야?
<|assistant|>

The model is post-trained to see this pattern and generate the next assistant response. Slapping an instruction template onto a base model does not make it behave like a chat model. Conversely, feeding a chat model a format different from the template used during training degrades performance. If you think the model is reading "meaning," this behavior is hard to explain. The model reads token patterns.

A Poor Tokenizer Makes Korean Expensive

When using LLM APIs, Korean often feels more expensive than English or burns through context faster. The tokenizer is why. When a Korean sentence gets split into more tokens than its English equivalent, input costs, output costs, KV cache memory, and attention computation all increase.

Consider these two sentences:

English: I will study tokenizers today.
Korean:  오늘은 토크나이저를 공부할 것이다.

With some tokenizers, the English sentence comes in around 7 tokens, while the Korean sentence gets split into more pieces due to postpositions and verb endings. With other tokenizers, Korean subwords are well-represented and the gap narrows. Token count directly affects serving cost, independent of language quality.

This is why tokenizer fertility matters when building Korean-language services. Fertility is roughly a measure of how many tokens are produced relative to the number of characters or words. For an accurate comparison, you need to encode the same evaluation corpus with each model's tokenizer and measure directly. Rather than vaguely claiming a model "handles Korean well," running 1,000 real user queries through each tokenizer and comparing average token counts gives a much more actionable answer.

texts = [
    "오늘은 토크나이저를 공부할 것이다.",
    "사용자의 결제 내역을 조회하고 환불 가능 여부를 판단한다.",
]

for text in texts:
    ids = tokenizer.encode(text)
    print(text, len(ids), ids[:20])

This code is simple, but it's surprisingly useful when selecting a model. For systems with long inputs — Korean customer support, document summarization, RAG — differences in average token count translate directly into GPU costs.

A Larger Vocabulary Is Not Always Better

A larger vocabulary lets more strings be represented as a single token, which shortens input sequences. That's appealing when you want good coverage of Korean, code, or domain-specific terminology.

The trade-off is that the embedding matrix and output projection grow accordingly. A language model computes next-token probabilities over the entire vocabulary, so a larger vocabulary means a larger final logits dimension. In today's large models, attention and MLP account for most of the compute, but vocabulary size still affects memory usage and training stability.

A vocabulary that is too large creates many rare tokens. Any token that does not appear frequently enough during training ends up with a weak embedding. On the other hand, a vocabulary that is too small forces every sentence to be fragmented into tiny pieces, wasting context. Good tokenizer training is about implementing the principle "keep frequent fragments large, keep rare fragments small" in a way that matches the data distribution.

When building domain-specific models, the temptation arises to add new tokens — medical codes, internal acronyms, long function names, product names — each as a single token. This decision deserves caution. Adding a new token creates a new embedding row that the existing model has never seen. You can train that embedding through fine-tuning, but anchoring it stably in the existing semantic space with limited data is difficult.

If domain terminology is long enough that tokenization costs are genuinely severe, extending the tokenizer may be worthwhile. For general domain knowledge injection, RAG or prompt design should come first. Changing the tokenizer touches the entire input distribution of the model.

Seeing It Directly Builds Intuition Fast

With Hugging Face transformers, you can immediately see how tokenizers differ across models.

from transformers import AutoTokenizer

models = [
    "gpt2",
    "bert-base-uncased",
]

text = "오늘은 SentencePiece와 Byte-level BPE를 공부한다."

for name in models:
    tok = AutoTokenizer.from_pretrained(name)
    ids = tok.encode(text)
    pieces = tok.convert_ids_to_tokens(ids)
    print("\n", name)
    print("tokens:", len(ids))
    print(pieces)

Running this shows that the same sentence is tokenized completely differently depending on the model. Some tokenizers stretch Korean characters into long sequences of byte-like fragments; others capture them more naturally as subwords. The differences become even more pronounced with code and URLs.

You can also reproduce the BPE training process at a small scale using the tokenizers library.

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

corpus = [
    "low lower lowest",
    "newer wider lowest",
    "token tokenizer tokenization",
]

with open("tiny.txt", "w", encoding="utf-8") as f:
    for line in corpus:
        f.write(line + "\n")

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(vocab_size=50, special_tokens=["[UNK]"])

tokenizer.train(["tiny.txt"], trainer)

encoded = tokenizer.encode("lowest tokenizer")
print(encoded.tokens)
print(encoded.ids)

The corpus here is far too small to be comparable to a real LLM tokenizer. Even so, it quickly builds the intuition that "fragments that appear together frequently end up in the vocabulary." Change the corpus and the merge results change too. A tokenizer is not a universal law of natural language — it is built from the statistics of its training corpus.

Following the Transformer Starts with Tokens

Open the Transformer paper and attention, positional encoding, and feed-forward networks jump out at you first. Trace through an implementation and the tokenizer comes before all of that, because the very first tensor fed into the model is the token ID sequence.

raw text
-> tokenizer.encode()
-> input_ids
-> embedding(input_ids)
-> transformer blocks
-> logits over vocabulary
-> next token sampling
-> tokenizer.decode()
-> raw text

The tokenizer is also at the end of a generative model. The model outputs a logit vector of size equal to the vocabulary, sampling picks one token ID from that distribution, and the sentence a human reads is the decode result. The tokenizer sits on both ends of the pipeline.

When we get to attention, query, key, and value appear seemingly out of nowhere. The starting point of Q, K, and V is token embeddings. The starting point of token embeddings is the ID sequence produced by the tokenizer. That is why Part 1 starts here.

The next post will cover embeddings — how a token ID becomes a vector, and why similar tokens end up in similar regions of the space. Working through that step by step makes attention feel a lot less sudden.