Elastic Training

If a job must have exactly 1,024 accelerators, losing one stops it. Elastic training lets the run continue on what is left — narrower, slower, but moving — and widen again when capacity returns. It turns a hard failure into a performance change, which is a much easier thing to survive.

The obstacle is that the batch size is baked into the hyperparameters. Shrink the worker count and either each worker takes more of the batch, which may not fit in memory, or the global batch shrinks, which quietly changes the optimisation problem you were solving. Systems that do this well hold the global batch fixed by raising gradient accumulation steps, so the maths of the run is untouched and only the wall clock moves.

The payoff is largest exactly where the failure rate is worst. A rigid job loses everything since its last checkpoint on every failure; an elastic one loses only the reconfiguration, so the share of work preserved rises with how often failures arrive. On preemptible capacity, where instances are reclaimed rather than broken, this is the difference between a job that finishes and one that never gets far enough between preemptions to reach the next checkpoint.