Reasoning RL

Rather than prompting a model to reason, train it to, by rewarding trajectories that reach verifiable answers. The surprise was how much emerged without being asked for: models trained this way lengthen their own reasoning, start checking their work, and backtrack when a line fails — behaviours nobody wrote a reward for.

The setup needs a domain where correctness is checkable automatically, which is why mathematics and code came first — the reward is a test suite or a numeric comparison, and no human sits in the loop. That constraint is the whole design: it is what makes the reward cheap enough to run at scale and what limits the approach to problems with a checker, which is a much smaller set than the set of problems people care about.

Reward on the outcome alone is sparse, and sparse reward on long trajectories is the hardest credit assignment there is — a correct final answer says nothing about which of two hundred tokens mattered. What makes it tractable is that the policy already generates plausible trajectories, so the search is not over an empty space but over a distribution already concentrated near solutions. That is why this works on a pretrained model and would not work from scratch.