Optimization / Schedules
verifiedLearning-Rate Schedule
A good learning rate early is a bad one late. Early on the model is far from anything useful and can take large steps; near the end it needs small ones to settle. A schedule moves the rate down over the run, usually along a cosine, so the same run gets both.
Cosine decay to near zero over the planned number of steps is the common default. The schedule is tied to the total length, so stopping early leaves the model at a rate it was never meant to finish at, and extending a run means the tail of the schedule has already been spent.
η sub t = η sub min + ½(η sub max − η sub min )(1 + cos(πt/T)) for total steps T. Because T appears in the formula, the trajectory for a run of length T is not a prefix of the trajectory for a longer one.
Loss over 1000 training steps, starting near 6.0. It falls to about 1.77, with 82% of the total improvement arriving in the first half. A second line shows flat rate, ending higher at about 2.16.
Loss under a decaying schedule against a flat rate. Drag the decay to watch the schedule pull the run further down in its final stretch.
Reviewed by opendroid · 2026-08-04
- arXiv:1608.03983 — SGDR: Stochastic Gradient Descent with Warm Restarts