Gradient Accumulation

Sometimes the batch size a run needs does not fit in memory. Gradient accumulation runs several smaller batches, adds their gradients together, and only then takes a step — the same update as one large batch, spread over several forward passes.

Skip the optimizer step for k micro-batches and divide the accumulated gradient by k. It buys effective batch size at the cost of wall-clock. The usual caveat — that a normalization layer computing statistics per micro-batch sees a smaller batch than intended — is about batch normalization, and does not apply to a transformer: layer norm and RMSNorm normalize across features within one position, so they never look at the batch at all.

g = (1/k)Σ sub _i=1..k  ∇L sub _B sub i  , then one step. Mathematically identical to a batch of size k|B| for the gradient, and for a transformer identical outright, since nothing in the forward pass computes a statistic over the batch.