Continuous Batching

Batching requests together uses the hardware well, but sequences finish at different times, and a batch that waits for its slowest member leaves most of the accelerator idle. Continuous batching lets a finished sequence leave and a waiting one take its place at the next step, so the batch is refilled constantly rather than assembled once.

Schedule at the granularity of an iteration rather than a request. A completed sequence frees its slot immediately and a queued one joins mid-flight, which raises throughput several times over static batching at the same latency. It needs a KV cache that can be allocated and released per sequence, which is what makes paged attention its natural partner.

With static batching of size B and lengths n sub _i , the batch occupies max sub _i  n sub _i  steps while doing Σ sub _i  n sub _i  tokens of useful work, so utilisation is Σn sub _i /(B·max n sub _i ) — poor whenever lengths vary. Iteration-level scheduling removes the max from the denominator, and utilisation stops depending on the spread of output lengths.