Attention
13 concepts, 13 reviewed and merged.
Behaviour
Core
- AttentionA model reading a sentence needs to decide, for each word, which other words matter.verified
- Causal MaskingA model trained to predict the next word must not be allowed to see it.verified
- Cross-AttentionCross-attention is attention between two different sequences.verified
- Multi-Head AttentionOne attention pass has to settle on a single set of weights, so it averages away anything it cannot decide between.verified
- Query, Key, ValueAttention borrows its vocabulary from databases.verified
- Self-AttentionSelf-attention is attention pointed at its own input: the queries, keys and values all come from the same sequence.verified
Efficiency
- FlashAttentionStandard attention writes the whole score matrix to memory, then reads it back to apply the softmax, then reads it…verified
- Grouped-Query AttentionMulti-head attention gives every head its own keys and values; multi-query gives them all one set.verified
- Multi-Query AttentionAttention heads each keep their own keys and values, and at generation time all of them sit in memory.verified
Position
- ALiBiALiBi drops positional encoding entirely and instead penalises attention for reaching far.verified
- Positional EncodingAttention has no idea what order anything is in.verified
- Rotary Position EmbeddingRather than adding position to a token's representation, RoPE rotates it — by an angle proportional to where it sits.verified