Throughput

How much work finishes per second, as opposed to how long any one piece takes. A serving system can be excellent at one and dismal at the other, and improving one usually costs the other — choosing where to sit on that trade is most of what serving engineering is.

The unit is where the arguments happen and it is usually left vague. Tokens per second across all requests, requests per second, and tokens per second for one user are three different quantities, and a system tuned for the first can be miserable on the third. Batching is the mechanism behind all of it: a larger batch raises aggregate tokens per second and raises what each request waits, so a throughput number quoted without the batch size and the latency it bought has not said anything.

Throughput rises with batch size only while the work is memory-bound: the weights have to be read either way, so extra requests ride the same read and the step takes barely longer — which is why batching is nearly free there, latency included. Once the arithmetic per byte loaded saturates the accelerator that stops, and step time grows in proportion to the batch: throughput flattens while latency begins rising linearly. So the trade is not smooth, and past saturation it is all cost. The batch size where it turns is the one number worth measuring, and it belongs to the model and the accelerator together rather than to either alone.