Scalable Oversight

Every method that trains a model on human judgement assumes a human who can judge. That holds while the task is one a person can check — is this summary faithful, is this answer polite — and stops holding the moment the model is doing something the supervisor cannot evaluate. Scalable oversight is the problem of supervising a system on tasks you cannot check yourself.

The proposals share a shape: use the model to help the human judge. Ask it to critique its own output, to debate another copy of itself with a human deciding, or to decompose a task until the pieces are checkable. Measuring whether any of this works is harder than proposing it, because the interesting case — supervision of something genuinely beyond the supervisor — is exactly the case that cannot be scored. Studies use a sandwich instead: a weak supervisor, a strong model, and a ground truth the researchers hold back.

The share of a system's outputs a supervisor can verify falls as capability rises, and that is the whole difficulty stated as a quantity. While it is high, ordinary preference learning works and the unverifiable remainder is a rounding error; as it falls, the training signal comes to rest on judgements nobody checked. Nothing about the methods changes at the crossing — what changes is that the assumption underneath them stops being true, silently.