Context Compression

If the window is expensive and much of what goes into it is redundant, spend some compute shortening the context before the model reads it. Summarise old turns, drop passages the query does not touch, replace verbatim documents with extracted facts. The context that arrives is smaller and, if the compression was any good, says the same things.

The methods split by what they throw away and when. Extractive selection keeps original tokens and drops the rest, which is safe and coarse. Abstractive summarisation rewrites, which is compact and can silently invent. Learned soft-prompt compression maps a span to a handful of vectors, which is the most compact and the least inspectable — you cannot read what was kept. Which risk is acceptable depends entirely on whether a wrong answer is recoverable downstream.

Compression is lossy against the query, not against the text, and that is the distinction methods get wrong. A summariser optimising for coverage of the document discards exactly the incidental detail a specific question turns on, and it does so more aggressively the harder it compresses. So the useful measure is not a compression ratio but the share of questions still answerable afterwards — which falls faster than the ratio does, because the questions lost are not a random sample.