Systems / Memory
verifiedOptimizer State
A model being trained takes far more memory than the same model being used. Adam keeps two extra numbers per parameter, and a full-precision master copy adds a third — so the weights are often less than a third of what training actually holds.
For Adam in mixed precision the standard accounting is 16 bytes per parameter: 2 for the bf16 weight and 2 for its gradient, then 4 for the fp32 master copy and 4 each for the two moments. That is why a model that infers comfortably on one accelerator needs several to train, and why sharding the state was the first thing distributed training attacked.
Per parameter: 2 bytes bf16 weight + 2 bf16 gradient + 4 fp32 master + 4 momentum + 4 variance = 16, the 2Ψ + 2Ψ + KΨ of the ZeRO accounting at K = 12. A 7B model therefore needs roughly 112 GB of state before activations, against 14 GB to serve it in bf16.
state-bytes holds 67% of the budget; rest holds the remaining 33%.
Bytes per parameter held by the state around a weight against the weight itself. Drag the state up to the fourteen this node counts and the weight is an eighth of the total.
Reviewed by opendroid · 2026-08-04
- arXiv:1412.6980 — Adam: A Method for Stochastic Optimization
- arXiv:1804.04235 — Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- arXiv:1910.02054 — ZeRO: Memory Optimizations Toward Training Trillion Parameter Models