Autoregressive Decoding

A language model does not write a sentence, it writes a token and then reads what it wrote. Each step produces a distribution over the vocabulary, one token is chosen from it, and the choice becomes part of the input for the next step. The whole output is that loop, run until something stops it.

Every step is a full forward pass for a single token, which makes generation memory-bound rather than compute-bound: the weights are read in their entirety to produce one token. That imbalance is why batching helps so much, and why speculative methods that verify several tokens per pass are worth the complexity.

x sub _t  ~ p(· | x sub _<t ) with p from a softmax over logits, optionally temperature-scaled or truncated by top-k or nucleus sampling. Generating n tokens costs n sequential forward passes regardless of hardware width, so latency is bounded by depth rather than throughput.