Batch Inference

Score a large set of inputs offline, with nobody waiting. It is the mode most machine learning actually runs in — nightly recommendations, document classification, embedding a corpus — and it is a different engineering problem from serving a request, because the thing being optimised is throughput and there is no latency to protect.

Which changes almost every decision. Batches can be as large as memory allows rather than as large as the latency budget permits. Preemptible capacity becomes attractive, because a failed shard is re-run rather than a failed request. Quantisation is easier to justify. And the system that does this well is a data pipeline with a model in it rather than a service — the hard parts are partitioning, retries and idempotency, none of which a serving stack spends much thought on.

Throughput per unit cost is the objective and it is bounded by the same arithmetic intensity as anywhere else, except that batch size is free to rise until memory stops it. Since the per-token cost of reading weights amortises across the batch, the marginal cost of an item falls steeply and then flattens once the weights are no longer the dominant term — so the useful batch size is the knee of that curve, and beyond it larger batches buy latency nobody is paying for.