Rollback

The fastest fix for a bad model is the previous one. Rollback is the ability to get back to it in minutes rather than hours, and it is worth more than almost any amount of pre-release testing — because testing reduces how often you are wrong, while rollback reduces how long being wrong costs you.

It is harder for models than for code because of state. A model that has been writing to a cache, populating a feature store, or generating training data for its own successor has left traces that reverting the binary does not undo. So a rollback plan has to name what the model wrote as well as what it served, and the systems that roll back cleanly are the ones that decided in advance which of a model's side effects are reversible.

Damage is the rate of bad requests times how long they lasted, and that duration is detection plus decision plus revert. Only the last of those is an engineering problem, and it is usually the smallest — a revert that takes two minutes sits behind a detection that took forty and an argument that took twenty. Teams that invest only in faster reverts are optimising the term that was already small.