Optimization / Generalization
verifiedScaling Laws
Loss falls predictably as models, data and compute grow — smoothly, along a straight line on a log-log plot, over many orders of magnitude. That predictability is what makes it possible to justify a training run costing millions before it starts: the result can be extrapolated from small ones.
Fit a power law in parameters, dataset size and compute; the exponents are small, so each factor of two in loss reduction costs far more than the last. The laws describe pretraining loss, not capability, and the relationship between the two is neither smooth nor predicted by them.
L(N) ≈ (N sub c /N) super α with α around 0.076 for parameters, and similar forms in data and compute. Each is measured with the other resources non-limiting, which is why the joint form matters more than any single curve.
Loss over 1000 training steps, starting near 6.0. It falls to about 2.15, with 88% of the total improvement arriving in the first half.
Loss against training compute for one model size. Drag the parameter count to watch the whole curve shift down along the power law.
Reviewed by opendroid · 2026-08-04
- arXiv:2001.08361 — Scaling Laws for Neural Language Models