Query, Key, Value

Attention borrows its vocabulary from databases. Each position emits a query — what am I looking for — and every position also offers a key, advertising what it has. Queries and keys are matched to decide where to look; the value is what actually gets read once the match is made. Splitting the three lets a position advertise something different from what it delivers.

Three learned projections of the same input: Q = XW sub _Q , K = XW sub _K , V = XW sub _V . Keys and queries must share a dimension because they are dotted together; values need not, and their dimension sets the output width. Separating K from V is what lets the model be found for one reason and read for another, which a single shared projection cannot express.

For input X ∈ ℝ super ^{n×d sub _model}  , W sub _Q , W sub _K  ∈ ℝ super ^{d sub _model ×d sub _k}   and W sub _V  ∈ ℝ super ^{d sub _model ×d sub _v}  . In self-attention all three project the same X; in cross-attention Q comes from one sequence and K, V from another. The parameter cost is d sub _model ·(2d sub _k  + d sub _v ) per head.