Cross-Attention

Cross-attention is attention between two different sequences. The queries come from the sequence being written, the keys and values from the one being read. It is how a decoder consults an encoded input, and how a model conditions on an image, an audio clip, or a retrieved document.

Identical arithmetic to self-attention with K and V projected from the other sequence. No causal mask is needed on the source, since the whole of it is available. Because the source encoding does not change while decoding, its keys and values are computed once and cached for every step.

CrossAttn(Y, H) = softmax(YW sub _Q (HW sub _K )ᵀ / √d sub _k )·HW sub _V , with Y the target sequence and H the source encoding. The score matrix is n sub _target  × n sub _source , so scoring costs Θ(n sub _target ·n sub _source ) against the Θ((n sub _target  + n sub _source )²) of self-attention over the two sequences concatenated.