Failure Recovery

At the scale large models are trained, hardware failure is not an incident — it is the weather. A single accelerator that dies takes the whole synchronous job with it, because every worker is waiting on a collective that will now never complete. Recovery is the loop that notices, evicts the dead node, restarts from the last checkpoint, and gets back to the step it was on.

The published logbooks are the honest source here. Llama 3's 16,000-GPU run reports a failure roughly every three hours, most of them a single accelerator or its host, and the OPT and BLOOM logs describe the same rhythm at smaller scale. The engineering that matters is detection latency and restart cost, not failure rate — you cannot buy hardware that does not fail, and you can absolutely build a job that restarts in ninety seconds instead of forty minutes.

Independent components make this arithmetic unforgiving: if one node fails on average every t hours, a job spanning n of them expects a failure every t/n hours. Ten thousand nodes each lasting a year between failures gives a job that breaks about every fifty minutes. The lost time per failure is detection plus restart plus the steps replayed since the checkpoint, and only the last of those three is bounded by the checkpoint interval — which is why teams that only tune checkpoint frequency stop making progress.