Overfitting

A model that keeps improving on its training data while getting worse on everything else has stopped learning and started memorising. The gap between the two curves is the whole diagnostic, which is why nobody trains without holding some data back.

Classically the fix is to stop where validation loss turns up. Large language models mostly do not get there — trained for a single pass over an enormous corpus, they are underfitted rather than overfitted, and the classical picture is further complicated by double descent, where test error falls again past the point where the old story says it should not.

Training risk falls monotonically with capacity while test risk is U-shaped in the classical regime. Beyond the interpolation threshold test risk can fall a second time, which the bias-variance decomposition alone does not predict.