Weight Averaging

Average the weights a single run passes through, rather than the weights of separate runs. Late in training a model does not converge to a point so much as wander around one, and the centre of that wander is usually a better model than any point on the path — including the last one, which is what you would otherwise ship.

The two forms differ in what they weight. A running average over late checkpoints treats them equally and needs a schedule that keeps the learning rate high enough to keep moving. An exponential moving average weights recent steps more and can run throughout training, which is why it is the default in diffusion training and in most self-supervised setups. Neither costs a forward pass; both cost one extra copy of the weights in memory.

This is the same variance argument as a soup, applied along one trajectory instead of across several. Successive iterates are correlated, so the effective number of independent samples is far below the number of checkpoints averaged — which is why the gain saturates quickly with the averaging window and why extending it further buys almost nothing. It also lands the result in a flatter region than any single iterate, which is the connection to Sharpness.