Inference / Memory
verifiedQuantized KV Cache
Store the attention cache in fewer bits. At long context the cache outgrows the weights, so this is where the memory actually is — and unlike the weights, it grows with every request rather than being paid once.
It quantises worse than weights do, and the reason is specific: the key and value tensors carry outliers concentrated in particular channels, so a single scale across a tensor is dominated by a few large values and everything else loses most of its range. Per-channel scaling for keys and per-token for values follows from that asymmetry rather than from tuning, and it is why a technique that works at four bits on weights needs more care here.
The cache is proportional to context length while the weights are fixed, so there is a length past which halving the cache's precision saves more memory than halving the weights' — and at long context it is not close. That crossing is what makes this worth separate engineering: below it the weights dominate and this is a rounding error, above it the cache is the whole problem and nothing else moves the number.
cache-bytes holds 17% of the budget; rest holds the remaining 83%.
Memory held by the attention cache, against the memory held by the weights, in equal units. Drag the cache up — it is context length that moves it in practice — to watch it overtake the weights; past that crossing it is the only term worth quantising.
Reviewed by opendroid · 2026-08-18
- arXiv:2401.18079 — KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- arXiv:2309.06180 — Efficient Memory Management for Large Language Model Serving with PagedAttention