Loss Spike

A long run is going well and then the loss jumps — sometimes recovering on its own over a few thousand steps, sometimes never. Large runs hit these repeatedly, and because a run costs millions of dollars, what to do about a spike is a real operational decision rather than a curiosity.

The standard response is unglamorous and effective: rewind to a checkpoint before the spike, skip a few hundred batches of data, and resume. PaLM's authors report doing exactly this, and note that the spike does not reappear when the same data is fed to the earlier checkpoint — so it is an interaction between a particular model state and a particular batch, not a poisoned example you could find and delete. Preventive measures act on the same interaction: gradient clipping, a longer warmup, and tighter control of the numerics.

The cost of a spike is not the spike; it is everything since the last good checkpoint. Rewinding discards every step between the checkpoint and the detection, so the replayed fraction is set by two things a team controls — how often it checkpoints, and how long the loss is allowed to climb before someone or something reacts. Detection latency is the term people forget, and on a run that is monitored by a human reading a dashboard each morning, it is measured in hours.