Data / Foundations
verifiedData Scaling Laws
Model quality improves predictably with data, and the improvement is a power law rather than a wall — which is what makes it plannable. The interesting regime now is being data-constrained rather than compute-constrained: the highest-quality text is finite, and repeating it works for a few epochs and then stops.
Repeated data is not free but it is close to free at first — up to roughly four epochs, repeated tokens are worth nearly as much as fresh ones, and past that they contribute almost nothing while still costing compute. That single result changes how a data budget is planned, because it turns 'we are out of data' into a bounded amount of extra headroom rather than a hard stop.
Loss falls as a power law in unique tokens, with repeated epochs decaying in value: the effective token count grows sublinearly with epochs and saturates. Combined with compute-optimal scaling this makes the joint problem a constrained optimisation over parameters, unique data, and epochs — not one where each is set independently.
Loss over 2000 training steps, starting near 7.2. It falls to about 1.94, with 93% of the total improvement arriving in the first half.
Loss against training progress at a fixed data budget. Drag the model size up to watch the floor fall — and watch how little the last doubling buys, which is what a data constraint looks like from inside a training run.
Reviewed by opendroid · 2026-08-18
- arXiv:2305.16264 — Scaling Data-Constrained Language Models