Optimization / Generalization
verifiedCompute-Optimal Training
Given a fixed compute budget, you can spend it on a bigger model or on more data. For years the field chose bigger models, and it was wrong: for a given budget the two should grow roughly in step, which means most large models of that era were badly undertrained.
Roughly 20 training tokens per parameter is the ratio the Chinchilla work identified as optimal for pretraining loss under a fixed budget. Serving cost changes the calculus — a smaller model trained on more tokens is cheaper to run forever after — so production models deliberately overshoot the ratio.
Minimising L(N, D) subject to C ≈ 6ND gives N ∝ C super 0 .5 and D ∝ C super 0 .5, so both scale as the square root of compute. The earlier convention scaled N far faster than D and landed away from that optimum.
Loss over 1000 training steps, starting near 6.0. It falls to about 2.16, with 88% of the total improvement arriving in the first half.
Loss over a run at this model size. Drag the parameter count to watch the floor fall — the budget a bigger model burns through faster is not in this curve, and it is what puts the optimum in the middle.
Reviewed by opendroid · 2026-08-04
- arXiv:2203.15556 — Training Compute-Optimal Large Language Models