AdamW

AdamW is Adam with the weight decay applied separately from the gradient rather than folded into it. That sounds like a detail and is not: under Adam's per-parameter scaling, decay folded into the gradient gets divided differently for every parameter, so it does not do what was configured. Separating it makes the setting mean what it says.

Apply the Adam update, then shrink the weights by η·λ·θ independently. The change is a few lines and it is the default for transformer training. Comparisons between papers that predate it and papers that do not are frequently not comparing the same regulariser at all.

θ ← θ − η(m̂/(√v̂ + ε) + λθ) is the coupled form; θ ← θ − η·m̂/(√v̂ + ε) − ηλθ is decoupled. The λθ term escapes the adaptive denominator, so the decay applied is uniform across parameters rather than inversely scaled by gradient variance.