Inference / Memory
verifiedContext Window
A model can only attend to what fits in its context window — the maximum number of tokens it can hold at once. Everything outside is invisible, not merely deprioritised. Extending it is one of the more expensive things to ask for, because attention cost grows with the square of the length and the cache grows linearly with it.
The limit comes from two places: positional encodings trained only up to a certain length, and memory for the score matrix and the cache. Extension methods attack these separately — rescaling the rotary base for the first, and paging or eviction for the second. A model run beyond its trained length usually degrades quietly rather than failing.
Attention cost is Θ(n²d) and cache memory Θ(n) per layer, so doubling the window quadruples one and doubles the other. Interpolating rotary frequencies by a factor s lets a model trained at n serve s·n with modest fine-tuning.
tokens-used holds 48% of the budget; rest holds the remaining 52%.
Tokens of conversation already in the window against the reserve held back for the reply. Drag the usage up to watch the conversation crowd everything else out — what no longer fits is not deprioritised, it is invisible.
Reviewed by opendroid · 2026-08-17
- arXiv:1901.02860 — Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
- arXiv:2004.05150 — Longformer: The Long-Document Transformer
- arXiv:2306.15595 — Extending Context Window of Large Language Models via Positional Interpolation