Attention / Efficiency
verifiedMulti-Query Attention
Attention heads each keep their own keys and values, and at generation time all of them sit in memory. Multi-query attention keeps the separate queries but shares one set of keys and values across every head, shrinking the cache by the number of heads at a small cost in quality.
Project queries per head as usual; project a single K and V shared by all heads. The cache shrinks by a factor of h, which is what makes long contexts affordable at generation time. Training from scratch with it loses a little quality, and models converted after the fact need a short adaptation phase.
head sub i = Attention(XW sub Q super i , XW sub K , XW sub V ) with W sub K and W sub V shared. Cache size falls from 2·h·d sub head ·n to 2·d sub head ·n per layer. Arithmetic intensity rises correspondingly, which is why the gain at generation time is larger than the parameter saving suggests.
heads-shared-away holds 97% of the budget; rest holds the remaining 3%.
Key-value cache that sharing removes against the single copy multi-query keeps, both as one head's cache. Drag the head count down to watch the saving collapse — almost all of it is already won by the first few heads shared away.
Reviewed by opendroid · 2026-08-04
- arXiv:1911.02150 — Fast Transformer Decoding: One Write-Head is All You Need