Optimization / Regimes
verifiedPretraining
Pretraining is the long, expensive first phase where a model learns language itself by predicting the next token over an enormous corpus. No labels, no task, no human supervision — just the text, and an objective that turns out to require learning a great deal in order to do well at.
One pass, or close to it, over trillions of tokens with cross-entropy on next-token prediction. It is where nearly all the compute and nearly all the capability come from; everything afterwards adjusts behaviour rather than adding knowledge. Data composition matters as much as data volume and is the least published part of any model.
Maximise Σ sub t log p sub θ (x sub t | x sub <t ) over the corpus. With C ≈ 6ND FLOPs for N parameters and D tokens, the budget is fixed before the run begins, which is why the schedule is planned against a step count decided in advance.
Loss over 1000 training steps, starting near 6.0. It falls to about 2.14, with 87% of the total improvement arriving in the first half.
Loss across a full pretraining run, falling fast and then slowly. Drag model size to watch the floor fall — how little arrives in the final stretch does not.
Reviewed by opendroid · 2026-08-04
- arXiv:2005.14165 — Language Models are Few-Shot Learners
- arXiv:2001.08361 — Scaling Laws for Neural Language Models