Warmup

At the very start of training the parameters are random and the gradients are large and meaningless. Taking full-size steps then can wreck the run in the first few hundred iterations. Warmup ramps the learning rate up from near zero over the first stretch, so the model gets its bearings before it starts sprinting.

Linear ramp over a few thousand steps, then hand off to the decay schedule. Post-norm transformers need it to train at all; pre-norm ones are stable without it and use it anyway because it still helps. Adam's second-moment estimate is also badly conditioned in the first steps, which is an independent reason for the ramp.

η sub _t  = η sub _max ·t/T sub _w  for t < T sub _w , then the decay schedule. The need is sharpest for post-norm, where the residual branch is not yet scaled and early updates can move the network far enough to be unrecoverable.