Inference / Memory
verifiedKV Cache
Generating a token requires attending to everything before it. Without a cache, each new token would recompute the keys and values for the entire prefix — work already done, and done again for every token. The cache keeps them, turning quadratic regeneration into a single step per token.
Store K and V per layer per head and append one column each step. Causal masking is what makes this valid: an earlier position's keys never change when a later one arrives. The cost moves from compute to memory, and the cache grows linearly with context and batch size until it, not the weights, is what fills the accelerator.
Cache size is 2 · layers · heads · d sub head · n · batch · bytes. Without it, generating n tokens costs Θ(n³d) in attention; with it, Θ(n²d). The memory is the reason multi-query and grouped-query attention exist.
context holds 50% of the budget; rest holds the remaining 50%.
Share of accelerator memory taken by the cache against the weights beside it. Drag the context length to watch the cache overtake the model itself.
Reviewed by opendroid · 2026-08-17
- arXiv:1911.02150 — Fast Transformer Decoding: One Write-Head is All You Need
- arXiv:2309.06180 — Efficient Memory Management for Large Language Model Serving with PagedAttention
- arXiv:2305.13245 — GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints