Adam

Different parameters need different step sizes, and nobody can tune them individually. Adam gives each one its own, derived from how large and how variable its gradients have recently been: parameters with small, steady gradients take bigger steps, and noisy ones take smaller. It is the default because it works without much tuning.

Track running averages of the gradient and of its square, correct both for initialisation bias, and divide one by the root of the other. The cost is two extra tensors the size of the model, which is why optimizer state rather than weights often dominates training memory. Its convergence proof was later shown to be flawed; it remains the default anyway.

m sub _t  = β₁m sub _t−1  + (1−β₁)g sub _t , v sub _t  = β₂v sub _t−1  + (1−β₂)g sub _t ², with bias-corrected m̂, v̂ and update θ ← θ − η·m̂/(√v̂ + ε). Defaults β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸.