Layer Normalization

Deep networks drift. As signals pass through many layers their scale creeps up or down, and training becomes a fight against the drift rather than against the problem. Layer normalization rescales each position's activations to a fixed mean and spread, so every layer receives input in the range it was tuned for.

Normalize across the feature dimension per position — not across the batch, which is what makes it work with variable-length sequences and batch size one. Placement matters more than it looks: post-norm, as originally published, needs learning-rate warmup to train at depth, while pre-norm trains stably without it and is what nearly every modern transformer uses.

LN(x) = γ ⊙ (x − μ)/√(σ² + ε) + β with μ and σ² computed over the d sub _model  features of a single position. γ and β are learned per-feature. Pre-norm places LN inside the residual branch, x + F(LN(x)); post-norm places it after, LN(x + F(x)).