Interpretability / Circuits
verifiedInduction Head
A pair of attention heads that together do something simple and powerful: find where the current token appeared before, and predict whatever followed it last time. It is copying with a lookup. It also appears abruptly during training rather than gradually, at the same moment a model's in-context learning ability jumps.
Two heads composed. A previous-token head writes the identity of position i−1 into position i; the induction head matches the current token against those writes and attends to the token just after the match. It is the most completely characterised circuit in a language model, and it is why in-context learning has a mechanism attached to it rather than only a benchmark curve.
Attention from query position t concentrates on position s+1 where the token at s matches the token at t — a match implemented in the QK circuit against the previous-token head's write, and a copy implemented in the OV circuit. The abruptness shows up as a visible bump in the training loss exactly where the two heads compose.
12 queries against 12 keys; a brighter cell means more of that query's attention went to that key. Each query sees itself and everything before it, and nothing after — the upper triangle is masked.
One head's attention over a causal window, under a penalty on distance. Drag the penalty up to watch the head collapse onto the token it just passed — an induction head is the opposite of this, ignoring distance to land on a match far behind it.
Reviewed by opendroid · 2026-08-18
- arXiv:2209.11895 — In-context Learning and Induction Heads