Optimization / Training
verifiedMomentum
Plain gradient descent forgets everything between steps, so it zig-zags across narrow valleys instead of running along them. Momentum keeps a running average of recent gradients and steps in that direction, cancelling the oscillation and accumulating speed where the direction is consistent.
Maintain a velocity buffer and update parameters with it rather than the raw gradient. It costs one extra tensor the size of the parameters and is the ancestor of every adaptive optimizer since. In transformer training it is rarely used alone, but the buffer it introduced is what Adam and its descendants build on.
v sub t+1 = βv sub t + ∇L(θ sub t ); θ sub t+1 = θ sub t − ηv sub t+1 . With β = 0.9 the effective step is roughly 1/(1−β) = 10 times the gradient along a consistent direction, while oscillating components cancel in the average.
8 values. The left group decays steeply; the right group is 99% of the way to flat, and reads flatter than the left.
Raw gradient magnitudes against the momentum-smoothed ones. Drag beta down to watch the averaging fail and the raw noise come back through.
Reviewed by opendroid · 2026-08-04
- arXiv:1609.04747 — An overview of gradient descent optimization algorithms