Evaluation: what the number means, and how it lies to you
Your model reports a validation loss of 2.31. Is that good? The question is harder than it looks, and answering it properly requires three things: a conversion into units that mean something, baselines that establish what "learned anything at all" looks like, and a clear-eyed understanding of the specific ways a good-looking number can be worthless.
Why the number must come from held-out data
A model with enough capacity can memorise its training set. Memorisation drives training loss toward zero while teaching the model nothing transferable, so training loss measures effort, not ability. The corpus is therefore split before training: the model fits one part and is measured on another it has never seen.
Strictly there are three roles. The training set is fitted. The validation set is checked repeatedly during development to choose hyperparameters and decide when to stop. The test set is looked at once, at the end, because every decision you make in response to a number leaks a little information about that data into your model. This campaign uses two splits, which is right for its scale — but the reason for the third is worth carrying: a validation set you have optimised against fifty times is no longer fully honest.
Three ways of stating the same number
Cross-entropy in nats is not intuitive to anyone. Two conversions make it concrete, and both are shown in the lab.
Perplexity
A perplexity of 1 is perfect prediction; a perplexity equal to the vocabulary size is total ignorance. If the model has perplexity 10 on character-level text, it is about as uncertain as someone picking uniformly among ten characters — which, against a 65-character alphabet, means it has learned a great deal. Perplexity turns an abstract loss into a number you can picture.
Bits per character
This is the information-theoretic reading, and it is exact: a model that assigns probability p to the observed text can encode it in −log₂ p bits. Language modelling and compression are formally the same problem. Raw ASCII English costs 8 bits per character; a good character-level model gets well under 2; Shannon's classic estimate for the entropy of English is around 1 bit per character. That number is a genuine target, not an arbitrary scale.
Baselines: what each one proves
A loss number in isolation says nothing. The lab computes three reference points from the same data, and beating each one proves something specific.
| Baseline | What it is | Beating it proves |
|---|---|---|
| Uniform | Every token equally likely: loss = ln(V) | The model has learned literally anything. A model above this is broken. |
| Unigram | Each token at its overall corpus frequency, ignoring context | The model has learned which tokens are common — frequency, not structure. |
| Bigram | Each token conditioned only on the single previous token | The model is genuinely using context beyond one token. This is the bar that matters. |
The bigram baseline is the interesting one. A bigram table is not a neural network at all — it is a counted frequency table, computable in one pass over the data, and on character-level English it is surprisingly strong. A transformer that fails to beat it has gained nothing from attention, embeddings, depth or gradient descent. The loss chart draws these as horizontal lines so you can see the moment your model crosses each one.
Overfitting, caused deliberately
Overfitting is usually taught as a diagram. It is much better understood as something you can produce on demand in under a minute, which is what the lab does: shrink the training set until there is less to learn than the model has capacity to memorise.
At 3% of the corpus, watch the two curves separate. Training loss keeps falling — smoothly, convincingly, better than it ever looked on the full corpus. Validation loss flattens, then turns and climbs. The generalisation gap between them is the diagnostic.
What to do about it
- More data. The only fix with no downside, and the reason frontier labs care so much about corpus size.
- Early stopping. Keep the checkpoint at the validation minimum rather than the last one. Nearly free.
- Regularisation — dropout, weight decay. Constrains what the model can express, at some cost to peak fit.
- A smaller model. Less capacity to memorise with. Effective, and the first thing to try when data is genuinely fixed.
Note what modern LLM pretraining implies about this list. Frontier models are trained for roughly one pass over an enormous corpus, so they rarely see the same token twice and classical overfitting barely arises during pretraining. It returns immediately during fine-tuning, where the dataset is small — which is Chapter 8's problem.
What loss still cannot tell you
Even a perfectly honest held-out loss is a proxy. It measures average next-token prediction over one corpus. It does not measure whether the model can add, reason, follow instructions, or avoid confidently inventing things — and small differences in loss can correspond to large differences in any of those. You saw the arithmetic version of this in Chapter 5: loss falling steadily while exact-match accuracy sat at zero.
That is why real evaluation is a stack, not a number: held-out loss for the training signal, task benchmarks for capabilities, and human or model-graded judgement for open-ended quality. Each layer catches failures the one below it cannot see.
Vocabulary
- Validation loss
- Cross-entropy measured on data the model was not trained on. The number that counts.
- Perplexity
- e^loss. The effective number of options the model is choosing between per token.
- Bits per character / byte
- loss / ln(2). The compression reading, and the tokenizer-independent way to compare models.
- Baseline
- A trivially-computed reference (uniform, unigram, bigram) that establishes what a given loss actually proves.
- Generalisation gap
- Validation loss minus training loss. Grows when the model starts memorising.
- Early stopping
- Keeping the checkpoint from the validation minimum instead of the final step.
Check yourself
If you can answer these without re-reading, the lab below will make sense. If you cannot, the relevant section is worth a second pass — that is a better use of your time than clicking buttons.
- Why is training loss an unreliable measure of quality, in one sentence?
- Convert a loss of 2.30 nats into perplexity and into bits per character.
- What does beating the bigram baseline prove that beating the unigram baseline does not?
- Why can two models' perplexities not be compared if they use different tokenizers?
- Describe the shape of both curves during overfitting, and say which one looks healthy.
- Give an example of a model capability that held-out loss cannot measure.
Where this comes from
- Shannon, "Prediction and Entropy of Printed English" (1951) — Where ~1 bit per character comes from.
- Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022) — Chinchilla: how loss actually responds to data and parameters.
- Zhang et al., "Understanding Deep Learning Requires Rethinking Generalization" (2016) — Networks can memorise pure noise — capacity alone explains nothing.