Architecture / Normalization
verifiedRMSNorm
RMSNorm is layer normalization with a step removed. It rescales activations by their root-mean-square without first subtracting the mean, on the observation that the recentring was doing less work than the rescaling. Fewer operations, one fewer statistic to compute, and in practice no loss in quality.
Divide by the root mean square of the features and multiply by a learned gain. There is no mean subtraction and usually no bias, which removes a reduction pass and a parameter vector per layer. The saving is small per call and large in aggregate, which is why it appears in most models trained after 2022.
RMSNorm(x) = γ ⊙ x / RMS(x) with RMS(x) = √((1/d)Σ sub i x sub i ²). Compared with LayerNorm the μ term is gone, so the transform is scale-invariant but not shift-invariant. β is typically dropped as well.
8 values. The left group decays steeply; the right group is 56% of the way to flat, and reads flatter than the left.
Feature magnitudes before and after rescaling by their root mean square. Drag the spread to watch inputs of very different scale arrive at the same normalized one.
Reviewed by opendroid · 2026-08-04
- arXiv:1910.07467 — Root Mean Square Layer Normalization