Prefix Caching

Many requests to the same system start identically — the same system prompt, the same tool definitions, the same document. Computing that prefix again for every request is work already done. Prefix caching keeps the attention state for a shared prefix and starts each request from where the shared part ended.

The distinction from KV Cache is the scope. That cache exists within one generation so each new token does not re-read the whole sequence; this one persists across requests so a prefix computed for one user serves the next. It needs the prefix to match exactly from the first token, which is why the ordering of a prompt is a serving decision — putting the variable part first destroys every hit, and moving it last costs nothing and recovers them all.

The saving is bounded by the prefix's share of the total prompt, so it is largest exactly where prompts are long and mostly shared — a long system prompt with a short user turn — and negligible where the prompt is mostly the user's. Multiply by the hit rate, which is a property of traffic rather than of the model: a thousand users of one agent share almost everything, and a thousand unrelated queries share nothing.