Batch Normalization

Normalise each feature across the examples in the batch, so every layer sees inputs with a stable scale however the layers below have drifted. It made deep networks trainable at depths that had not worked before, and it did so for reasons that are still argued about — the original explanation has largely been abandoned and the technique has not.

The difference from Layer Normalization is the axis, and everything follows from it. Normalising across the batch means each example's output depends on the other examples in its batch, which is fine in training and impossible at inference — so it keeps a running average to use instead, and that average is a second set of state that can be wrong. It also degrades as the batch shrinks, and fails at a batch of one. Layer normalisation crosses features within one example, has no batch dependence, and is why the transformer uses it.

The batch statistics are estimates, so their error scales as one over the square root of the batch size — which is why small batches destabilise it and why the training-time and inference-time computations genuinely differ rather than merely being implemented differently. That gap is the source of most batch-norm bugs: a model in the wrong mode is not slightly off, it is normalising by the wrong numbers entirely.