Evaluation / Metrics
verifiedPerplexity
Perplexity asks how surprised a model is by text it has not seen. If it were choosing uniformly among 20 words at every step, its perplexity would be 20. Lower means less surprised, which usually means better — but only on the corpus it was measured against, and two perplexities computed on different text or different tokenizers are not comparable at all.
The exponential of the mean cross-entropy per token. It is the metric pretraining actually optimises, which makes it the honest one for that stage and a poor one for anything after — an instruction-tuned model can get worse perplexity while becoming far more useful. Changing the tokenizer changes it even on identical text, so it travels badly between papers.
PPL = exp(−(1/N) Σ sub t log p(x sub t | x sub <t )), the exponentiated cross-entropy. Since it is per token, a vocabulary that splits words into more pieces lowers it without the model improving — the denominator N is a property of the tokenizer rather than of the text.
Loss over 1000 training steps, starting near 6.0. It falls to about 2.12, with 88% of the total improvement arriving in the first half.
Cross-entropy over a training run — perplexity is its exponential, so this curve is that number on a log scale. Drag model size to watch the floor fall.
Reviewed by opendroid · 2026-08-04
- arXiv:1312.3005 — One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
- arXiv:2101.00027 — The Pile: An 800GB Dataset of Diverse Text for Language Modeling