Speculative Decoding

Generating one token at a time wastes most of an accelerator: the model is enormous, the work per token is tiny, and the hardware sits idle waiting on memory. Speculative decoding has a small fast model guess several tokens ahead, then checks all the guesses in a single pass of the big one. Right guesses are kept, the first wrong one is corrected, and the run moves several tokens for the price of one.

A draft model proposes k tokens; the target model scores all k+1 positions in one forward pass, which costs about what one token cost before because decoding is memory-bound rather than compute-bound. Accepted tokens are exactly what the target would have produced — the output distribution is unchanged, which is what separates this from every other speed-up that trades quality for latency.

For draft distribution q and target p, accept a proposed token x with probability min(1, p(x)/q(x)); on rejection sample from the normalised residual max(0, p − q). The result is distributed exactly as p. Expected tokens per pass rises with the acceptance rate α as (1 − α super ^k+1 )/(1 − α), so a weak draft still helps and a good one helps a great deal.