Deployment / Practice
verifiedRegression Suite
A fixed set of cases the model must still get right before a new version ships. It is not a benchmark and it is not trying to measure quality — it is trying to stop a specific thing that broke once from breaking again, which is a much narrower and much more achievable goal.
Real suites are mostly scar tissue, and that is a feature. Every incident contributes the case that would have caught it, so the suite grows in the shape of the system's actual failure modes rather than of anyone's guess at them. The discipline that keeps it useful is that a case is added when an incident is closed, not when someone remembers — a suite assembled up front tests what the authors imagined and a suite grown from incidents tests what happened.
The limit is underspecification: two models scoring identically on the same fixed set can behave differently on everything outside it, and there are many more such models than the suite can distinguish. So a green suite is evidence about the cases in it and nothing beyond — it says the known failures did not recur, which is exactly what it was built to say and considerably less than passing it feels like.
incident-cases holds 17% of the budget; rest holds the remaining 83%.
Cases added after an incident, against the cases someone wrote up front, in cases. Drag the incidents up to watch the suite become mostly scar tissue — which is the healthy shape, because it grows toward the failures the system actually has.
Reviewed by opendroid · 2026-08-18
- arXiv:2011.03395 — Underspecification Presents Challenges for Credibility in Modern Machine Learning
- arXiv:2306.05685 — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena