Bias-Variance Tradeoff

A model can be wrong because it is too simple to capture the pattern, or because it is so flexible it fits the noise as well. Classically these trade off: reduce one and the other rises, with a sweet spot between them. Modern practice broke that story — past a certain size the error falls again, and the sweet spot turned out not to be the end of the curve.

Bias is error from the model class being too restrictive; variance is error from sensitivity to which training sample you happened to get. The classical U-curve is real and it is not the whole picture: at the interpolation threshold, where a model has just enough capacity to fit the training data exactly, test error peaks — and then falls again as capacity grows past it. Any intuition about "too many parameters" formed before that result is out of date.

E[(y − f̂(x))²] = Bias[f̂]² + Var[f̂] + σ², an exact decomposition for squared error. What it does not say is how capacity maps onto the two terms — double descent shows variance peaking at the interpolation threshold and falling beyond it, so the classical monotone reading was an assumption about that mapping rather than a consequence of the identity.