Inference Scaling Law

Accuracy improves predictably with compute spent at inference, in the same regular way it improves with compute spent at training — and the two are substitutes. That means a serving budget is a choice between a larger model and a longer-thinking one, and the choice has a right answer that depends on the problem rather than on taste.

The practical finding is that on problems a model can nearly do, extra inference compute beats extra parameters at equal cost, sometimes by a wide margin. On problems well beyond it, the reverse holds — thinking longer cannot find what the model has no representation for. So the useful question for a deployment is not which axis is better but where its problem distribution sits relative to that crossover, which is measurable and rarely measured.

The distinction from Scaling Laws is the resource being scaled. Training compute is paid once and amortised over every query; inference compute is paid per query, so its cost grows with traffic while the model's does not. A curve showing inference compute buying accuracy is therefore describing a recurring bill, and the two curves cannot be compared without fixing the query volume — which is why the crossover point moves with how much traffic a system serves.