Attention

A model reading a sentence needs to decide, for each word, which other words matter. Attention is that decision made numerically: every position scores every other position, the scores become weights, and the position reads a weighted blend of what it found. Nothing is hard-wired about which words matter — the weights are computed fresh for every input.

Score each query against every key, scale by the square root of the head dimension, softmax the scores into weights, and take the weighted sum of the values. The scaling is not cosmetic: without it the dot products grow with dimension, the softmax saturates, and gradients vanish. Cost is quadratic in sequence length because every position scores every other one.

Attention(Q,K,V) = softmax(QKᵀ / √d sub _k ) · V, with Q ∈ ℝ super ^{n×d sub _k}  , K ∈ ℝ super ^{m×d sub _k}  , V ∈ ℝ super ^{m×d sub _v}  . The softmax is row-wise, so each query's weights over the m keys sum to one. The √d sub _k  divisor keeps the variance of the dot products near one when the entries of Q and K are unit-variance.