Inference
20 concepts, 20 reviewed and merged.
Adaptive
- Adaptive ComputationSpend more on hard inputs than on easy ones.verified
- Early ExitStop partway up the stack when the answer is already clear.verified
- Mixture of DepthsLet each token choose how many layers it passes through.verified
- Model CascadeAsk a cheap model first.verified
- Model RoutingDecide which model should answer before any of them does.verified
- Semantic CacheReturn a stored answer when a new question means the same as an old one.verified
Decoding
- Autoregressive DecodingA language model does not write a sentence, it writes a token and then reads what it wrote.verified
- Beam SearchGreedy decoding takes the most likely next token every step and can walk into a corner, because a mediocre token now…verified
- Nucleus SamplingTop-k always keeps the same number of tokens, whatever the model thinks.verified
- Sampling TemperatureOne dial between always taking the most likely next token and picking almost at random.verified
- Speculative DecodingGenerating one token at a time wastes most of an accelerator: the model is enormous, the work per token is tiny, and…verified
- Top-k SamplingSampling from the full distribution occasionally picks something absurd, because the tail holds thousands of tokens…verified
Efficiency
Memory
- Context WindowA model can only attend to what fits in its context window — the maximum number of tokens it can hold at once.verified
- KV CacheGenerating a token requires attending to everything before it.verified
- Paged AttentionA KV cache has to be contiguous, and nobody knows how long a sequence will get, so servers reserve the maximum up…verified
- Quantized KV CacheStore the attention cache in fewer bits.verified