Weight Decay

A model with enormous weights has usually memorised something rather than learned it. Weight decay pulls every parameter slightly toward zero on every step, so a weight has to keep earning its size. It is the simplest regulariser and often the only one a large model needs.

Shrink each parameter by a fixed fraction per step. Added to the gradient it becomes L2 regularisation, which interacts badly with adaptive optimizers — the adaptive scaling divides the penalty differently per parameter, so the effective decay is not what was configured. Decoupling it from the gradient is what AdamW fixes.

Coupled: g ← g + λθ, then the optimizer scales both together. Decoupled: θ ← θ − ηλθ applied separately from the gradient step. Under Adam the two differ because the adaptive denominator √v̂ divides the coupled penalty per-parameter and the decoupled one not at all.