Attention Sink

Trained transformers dump a surprising amount of attention on the very first token, regardless of what it is. The softmax has to put its weight somewhere even when a position has nothing it needs, and the first token becomes the place it goes. Evict it from the cache and quality collapses, for a token carrying no information.

The mechanism is that softmax weights must sum to one, so a head with nothing to retrieve still distributes weight, and the earliest position is visible to every query. Streaming methods exploit this by keeping the first few tokens permanently while rolling the rest, which restores quality that naive window eviction destroys.

Because Σ sub _j  a sub _ij  = 1 by construction, a head cannot attend to nothing; the surplus concentrates on positions visible to all queries. Retaining k initial tokens alongside a sliding window of size w gives a cache of k + w rather than n while keeping perplexity near the full-context value.