Learning Rate

The learning rate decides how far to move on each step. Too small and training takes forever; too large and it overshoots and diverges. It is the single most important number in a training run, and the range that works spans about one order of magnitude.

Almost never held constant in practice. The workable maximum scales with batch size and shrinks as the model gets larger, and the standard way to find it is a short sweep rather than any principled derivation. A run that diverges in the first few hundred steps is almost always this parameter.

θ ← θ − η∇L. Stability requires η below roughly 2/λ sub _max  of the loss Hessian, which is neither known nor stationary. Linear scaling η ∝ |B| holds empirically over a wide batch range, breaking down at the top of it.