Tokenizer

Why a model cannot read text

The first layer of a language model is a lookup table: a matrix with one row per known symbol. To use it you need an integer index. So the job of the tokenizer is to define a finite alphabet of pieces — the vocabulary — and a reversible mapping between text and sequences of indices into it.

Reversible matters. Whatever the model eventually produces comes back out as token ids, and decode(encode(text)) must return the original text exactly, or the system leaks corruption no amount of training can fix. Every tokenizer in this chapter round-trips losslessly.

The three obvious options, and why two of them are bad

The sub-word compromise gives common words their own single token while rare words fall back to pieces, and — in the byte-level variants used by GPT-2 onward — the ultimate fallback is raw bytes, so nothing is ever unrepresentable. You get short sequences without an out-of-vocabulary cliff.

Byte-pair encoding, step by step

BPE was a compression algorithm from 1994, repurposed for tokenization in 2015. The training procedure is almost embarrassingly simple, which is a large part of why it won:

1. Start with the vocabulary being every individual character in the corpus.
2. Count every adjacent pair of tokens in the corpus.
3. Take the most frequent pair and add it to the vocabulary as one new token.
4. Replace every occurrence of that pair with the new token.
5. Repeat for a fixed number of merges. The merge count is the knob you control.

corpus: "low lower lowest low low"

start   l o w _ l o w e r _ l o w e s t _ l o w _ l o w
merge 1 ("l","o") -> "lo"      lo w _ lo w e r _ lo w e s t _ lo w _ lo w
merge 2 ("lo","w") -> "low"    low _ low e r _ low e s t _ low _ low
merge 3 ("low","_") -> "low_"  low_ low e r _ low e s t _ low_ low
merge 4 ("e","r") -> "er"      low_ low er _ low e s t _ low_ low

vocabulary grew by 4; the sequence went from 25 tokens to 13.

Notice what happened without anyone specifying it: the algorithm discovered the stem low and the suffix er purely from frequency. Nobody supplied a dictionary, a grammar or a notion of morphology. This is the whole trick — frequency in a large corpus is a surprisingly good proxy for linguistic structure.

Encoding new text applies the learned merges in the order they were learned. That ordering is part of the tokenizer: the merge list is the model. Real tokenizer files are little more than a vocabulary and an ordered list of merge rules.

What a bigger vocabulary buys, and what it costs

Every merge makes sequences shorter. Shorter sequences are pure profit for a transformer, because attention cost grows with the square of sequence length: a corpus compressed 2× costs roughly 4× less attention compute to model, and the same context window suddenly holds twice as much text.

The bill

Vocabulary size V appears twice in the parameter count, at both ends of the model. The embedding table is V × d, and the output layer that turns a final hidden vector back into a score per token is another d × V:

At GPT-2 scale (V = 50,257, d = 768) that is 77 million parameters spent on nothing but the alphabet — about 62% of the 124M model. The softmax over V also runs on every position of every forward pass, so vocabulary size is a per-token compute cost, not just a memory one.

There is a second, subtler cost: statistical dilution. A token that appears fifty times in the training set gets fifty gradient updates to its embedding row, which is not enough to learn a good vector. Push the vocabulary too large and the tail fills with tokens the model never really learns.

The three numbers that tell you if it worked

The lab below reports the same measurements you would use to evaluate a real tokenizer.

• Compression (characters per token) — the headline number. 1.0 means character-level. Real English tokenizers land around 4. Higher is better for a fixed vocabulary size; comparing compression across different vocabulary sizes is meaningless without also comparing the parameter cost.
• Held-out OOV rate — what fraction of text the tokenizer has to fall back on for text it was not fitted to. This is measured on a slice of the corpus the merge algorithm never saw, which is the only way the number means anything.
• Embedding + head parameters — the bill, computed as 2 × V × d. Watch it climb as you drag the merge slider, and note how quickly it starts dwarfing the rest of a small model.

Consequences you will meet later

Tokenization is chosen before training and is then frozen for the life of the model. Several famous LLM weaknesses are downstream of that fact rather than of the network at all:

• Arithmetic is hard partly because of tokenization. If 1234 is one token and 1235 is another, the model cannot see the digits it is supposed to be adding. Modern tokenizers deliberately split digits into groups to help.
• Character-level tasks are hard. "How many r's in strawberry?" is difficult for a model that never saw the letters — it saw two or three opaque sub-word chunks.
• Non-English text costs more. A tokenizer fitted mostly to English needs far more tokens per sentence in other languages: same meaning, more context consumed, more money per API call, worse effective memory.
• Fine-tuning cannot add tokens for free. New rows in the embedding table start as noise. In Chapter 8 you will fine-tune, and the instruction data has to stay inside the alphabet the model already knows.

Vocabulary

Token

The atomic unit a model reads and writes: an index into the vocabulary.

Vocabulary

The full set of tokens, fixed before training. Its size is written V.

Merge

One BPE step: the most frequent adjacent pair becomes a single new token.

Compression ratio

Characters per token. How much text one position of context buys you.

OOV (out of vocabulary)

Text the tokenizer cannot represent with learned pieces and must fall back on.

Byte-level BPE

BPE run over raw bytes, so any input at all is representable. Used by GPT-2 onward.

Check yourself

If you can answer these without re-reading, the lab below will make sense. If you cannot, the relevant section is worth a second pass — that is a better use of your time than clicking buttons.

1. Why does a word-level vocabulary fail on real text, even at 100,000 words?
2. Where does vocabulary size appear in the parameter count, and why twice?
3. If compression improves from 2.0 to 4.0 characters per token, what happens to the amount of text a fixed context window holds — and to the cost of attention over it?
4. Why is the OOV rate measured on held-out text rather than on the text the merges were learned from?
5. Give one LLM weakness that is caused by tokenization rather than by the network.

Where this comes from

• Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units" (2015) — The paper that brought BPE to NLP.
• Karpathy, "Let's build the GPT Tokenizer" — A byte-level BPE implementation built from scratch, in the same spirit as this chapter.
• Kudo, "Subword Regularization" / SentencePiece — The Unigram alternative to BPE, used by Llama and T5.