Selective Prediction

Give the model the option to say nothing. A classifier that answers everything at 92% may be less useful than one that answers four fifths of the time at 99% and hands the rest to a person — if the escalation is cheap enough. The design question is not how accurate the model is but where the threshold goes, and that is a question about the cost of being wrong rather than about the model.

Rank inputs by any confidence score, then abstain below a cutoff, and plot risk against coverage as the cutoff moves. The curve is the deliverable, not any single point on it: it tells the operator what accuracy is available at each level of automation, which is the number a deployment actually needs. A model with worse accuracy can have a better risk–coverage curve, and then it is the better model.

Coverage and risk fall together, but not at the same rate. The first examples you drop are the ones the model was least sure about, which are disproportionately the ones it got wrong, so early abstention buys a lot of accuracy for very little coverage — and the trade decays from there. The curve is only as good as the ranking underneath it, so a badly ordered confidence score produces a nearly straight line and no useful operating point anywhere on it.