Landscape / Generalization
verifiedDouble Descent
Test error falls as a model grows, then rises around the point where it can just barely fit the training data, then falls again — and keeps falling. The classical U-shaped curve is real and is only the left half of the picture. Bigger past the peak is better, which is the opposite of what parameter counting predicts.
The peak sits at the interpolation threshold, where capacity roughly equals dataset size, and it is the worst place to be — a model just large enough to fit the data has exactly one way to do it and no freedom to pick a good one. The same shape appears in epochs as well as parameters, which is why training longer can pass through a worse region before improving.
Past the interpolation threshold many zero-training-error solutions exist, and the optimiser's implicit bias selects among them — so error falls again for the same reason Implicit Regularization describes. At the threshold that freedom vanishes, which is why the peak is there rather than anywhere else.
capacity-beyond-threshold holds 13% of the budget; rest holds the remaining 87%.
Capacity past the point where the data can just be fitted against the capacity needed to fit it, in equal units. Drag past the threshold to watch the freedom to choose a good solution appear — at zero there is exactly one way to fit, and it is a bad one.
Reviewed by opendroid · 2026-08-18
- arXiv:1912.02292 — Deep Double Descent: Where Bigger Models and More Data Hurt