Self-Attention

Self-attention is attention pointed at its own input: the queries, keys and values all come from the same sequence. Every position gets to look at every other position in one step, so information moves across the whole sequence without being passed along hop by hop the way a recurrent network passes it.

Project the same input three ways and run attention. The property that matters is the path length between any two positions: one, regardless of distance. A recurrent network needs a number of steps proportional to the gap, which is what makes long-range dependencies hard to learn. The price is the quadratic score matrix, and every efficiency variant since has been an attempt to avoid materialising it.

SelfAttn(X) = softmax(XW sub _Q (XW sub _K )ᵀ / √d sub _k ) · XW sub _V . The score matrix is n×n for a sequence of length n, so both time and memory are Θ(n²d) before any fusion. Position information is absent by construction: permuting the rows of X permutes the output identically, which is why positional encoding exists.