Attention / Position
verifiedALiBi
ALiBi drops positional encoding entirely and instead penalises attention for reaching far. Each head subtracts a fixed amount from every score in proportion to the distance between the two positions, with different heads penalised at different rates. Near tokens are preferred by construction rather than by anything learned.
Add a static, non-learned bias of −m·|i−j| to the scores before the softmax, with a per-head slope m from a geometric sequence. No parameters, no embedding, and the model trained at one length runs at a longer one without retraining, which is the property it was built for.
S sub ij + m sub h ·(j − i) for head h, with slopes m sub h taken as a geometric sequence such as 2 super −8h/H . Heads with steep slopes attend locally; shallow ones reach further. The bias is added before the softmax so it scales the resulting weights multiplicatively.
10 queries against 10 keys; a brighter cell means more of that query's attention went to that key. Each query sees itself and everything before it, and nothing after — the upper triangle is masked.
One head's attention once ALiBi's linear penalty on distance is added to the scores. Drag the penalty up to watch the head pull in toward the tokens just passed — ALiBi hands each head a different slope, so the set of them covers near and far at once.
Reviewed by opendroid · 2026-08-04
- arXiv:2108.12409 — Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation