Inference / Serving
verifiedTail Latency
The slowest requests, not the typical one. A median of 200 milliseconds with a 99th percentile of eight seconds is a system that feels broken to one user in a hundred — and if a page makes ten calls, one page in ten contains one of them. The average hides exactly the experience people complain about.
In generation the tail has specific causes and each has a fix: a long prompt monopolising a batch slot, a queue that formed while the autoscaler was still cold, a request stuck behind another's prefill. The reason Disaggregated Serving, Continuous Batching and preemption exist is the tail rather than the mean, and a change that improves throughput by making the tail worse is common enough that reporting both is not optional.
Fan-out is what makes this brutal. If a page issues n independent calls, the chance all of them beat the 99th percentile is 0.99 super n — so ten calls means roughly one page in ten contains a tail request, and a hundred calls means two pages in three do. The percentile a service should target is therefore set by how many times a page calls it, and a service quoting a comfortable p99 to a caller that fans out widely has answered the wrong question.
fanout-calls holds 9% of the budget; rest holds the remaining 91%.
Pages carrying at least one tail request, against pages where every call was fast. Drag the fan-out up to watch the share climb — with one call in a hundred landing in the tail, ten calls already spoil about one page in ten, and the per-call percentile never moved.
Reviewed by opendroid · 2026-08-18
- arXiv:2309.06180 — Efficient Memory Management for Large Language Model Serving with PagedAttention
- arXiv:2403.02419 — Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems