Edge of Stability

Classical optimisation says a step size above two over the curvature makes gradient descent diverge. Neural network training routinely sits just past that line and does not diverge — the loss stops falling monotonically and oscillates while still making progress, and the curvature adjusts until it sits right at the boundary.

This is why the largest stable learning rate is where tuning ends up, and why loss curves at good hyperparameters are visibly non-monotone rather than smooth. It also explains warmup: early training has low curvature, so a rate that will be fine later is far past the boundary at step zero. A curve that looks too clean is often a run leaving speed on the table.

Descent on a quadratic is stable while η < 2/λ sub _max . Training instead drives λ sub _max  up until ηλ sub _max  ≈ 2 and then hovers there, so the sharpness is set by the learning rate rather than the other way round. The loss decreases over long windows while increasing over short ones, which is what the oscillation in the figure is.