Grouped-Query Attention

Multi-head attention gives every head its own keys and values; multi-query gives them all one set. Grouped-query sits between: heads are divided into groups, and each group shares. It recovers most of the quality lost by full sharing while keeping most of the memory saved.

Choose g key-value groups with h heads spread across them; g = h is multi-head, g = 1 is multi-query. Existing multi-head checkpoints can be converted by mean-pooling the projections within each group and briefly fine-tuning, which is why it spread quickly rather than needing new pretraining runs.

Cache size scales as 2·g·d sub _head ·n per layer, interpolating between the two endpoints. Quality is close to multi-head at g of roughly 8 for h of 64, so most of the memory reduction is available at little cost.