Human in the Loop

Some actions should not happen without a person saying yes — sending money, deleting data, emailing a customer. The design question is not whether to ask but when, because an agent that asks about everything is a slow way to do the task yourself, and one that asks about nothing is a liability.

Gate on reversibility and blast radius rather than on model confidence, which is not calibrated for this and never has been. Batching approvals defeats the purpose — a person clicking through twenty prompts is not reviewing them — so the rate of asking is itself a safety property. Approval fatigue is the failure mode, and it is caused by asking too often rather than too rarely.

An approval is worth its cost when the expected harm avoided exceeds the interruption, so the threshold should track harm rather than uncertainty. Since attention degrades with frequency, the value of each approval falls as the rate rises — meaning a system asking twice as often does not deliver twice the oversight, and past some rate delivers less in total.