Chain of Thought

Asked for the answer to a multi-step problem, a model that must produce it immediately often gets it wrong. Asked to work through the steps first, it does much better. The reasoning is written into the output because that is the only place the model has to put it — there is no scratchpad, so the tokens are the scratchpad.

Prompt for intermediate steps, either with worked examples or a bare instruction to think step by step. It helps on arithmetic and multi-step reasoning and barely at all on tasks a single step already solves. The gain grows with model scale, and the written chain is not necessarily the computation that produced the answer — it is a plausible one.

Generating m reasoning tokens before the answer costs m extra sequential forward passes, so accuracy is bought with latency at roughly the rate the chain is long. The model carries nothing between passes but the tokens it has already emitted and the keys and values they produced — there is no hidden scratch state — so a longer chain is literally more forward passes spent on the problem. Those passes get dearer as they go: with a KV cache, attention over a context of n costs Θ(n·d) per token, so the trade is worse than linear in m.