Process Supervision

Reward each step of the reasoning rather than only the final answer. It is more expensive to label — someone has to read every line — and it fixes a specific problem that outcome rewards create, which is that a model reaching the right answer through nonsense gets exactly the same credit as one that reasoned correctly.

Outcome supervision is cheap because the answer is checkable automatically; process supervision needs a human or a strong model to judge each step, so the datasets are small and expensive. What buys that cost is a much better verifier: a step-level reward model catches an error where it happens instead of inferring from a wrong answer that something, somewhere, went wrong. It also gives the model a signal it can act on, since "step four is wrong" is actionable and "your answer is wrong" is not.

The credit assignment problem is why the difference is large. With outcome-only reward, a trajectory of twenty steps gets one bit of feedback for all twenty, and a correct answer reached by a lucky cancellation reinforces every step that produced it — including the wrong ones. Step-level reward gives one signal per step, so the feedback a trajectory carries grows with its length instead of staying at one bit however long the reasoning gets.