Multi-Head Attention

One attention pass has to settle on a single set of weights, so it averages away anything it cannot decide between. Multi-head attention runs several passes side by side, each with its own projections, so one head can follow grammatical agreement while another tracks the subject of the sentence. The results are concatenated and mixed.

Split d sub _model  into h heads of dimension d sub _k  = d sub _model /h, run attention in each, concatenate, and project back with W sub _O . Total parameter count is unchanged from a single head of full width, so heads buy diversity rather than capacity. Empirically many heads are prunable after training, which suggests the gain is in optimisation as much as expressiveness.

MultiHead(X) = Concat(head sub ₁ ,…,head sub _h )W sub _O  where head sub _i  = Attention(XW sub _Q  super ⁱ , XW sub _K  super ⁱ , XW sub _V  super ⁱ ) and W sub _O  ∈ ℝ super ^{h·d sub _v ×d sub _model}  . With d sub _k  = d sub _v  = d sub _model /h the total cost matches single-head attention at full width, since the h smaller matmuls sum to the same work.