Data / Schedules
verifiedCurriculum
Show easy examples first and hard ones later, as a teacher would. The idea is old and intuitive and its record in deep learning is mixed — it helps clearly in some settings and does nothing in others, and which is which is not well predicted in advance.
Where it reliably helps is when the hard examples are hard because they depend on something the easy ones teach — long context after short, multi-step reasoning after single-step. Where it does not is when difficulty is just noise, since ordering noise is still noise. Modern large-scale training uses a weak version of this in length and domain scheduling rather than a strict easy-to-hard ordering.
Order examples by a difficulty measure and anneal the sampling distribution from easy toward the full one. The mechanism claimed is a smoother optimisation path — early steps see a simpler loss surface — which is a claim about the trajectory rather than the optimum, so any benefit is about reaching a good solution faster rather than reaching a better one.
easy-examples holds 50% of the budget; rest holds the remaining 50%.
Easy examples early in training against hard ones held back, in examples. Drag the curriculum up to watch the early distribution simplify — the claim is about how fast a good solution is reached, not about which one.
Reviewed by opendroid · 2026-08-18
- arXiv:2305.16264 — Scaling Data-Constrained Language Models
Origin · not linkable
- Bengio et al. 2009 — Curriculum Learning · ICML 2009 · doi:10.1145/1553374.1553380