Inference Accelerator

A chip built to serve a model needs a different shape from one built to train it. Training has to hold gradients and optimiser state and every activation on the forward path; serving needs the weights and very little else. That difference is large enough that the two workloads want different ratios of memory to arithmetic, different number formats, and different economics.

Serving silicon leans into it. Lower-precision integer units are acceptable because there is no backward pass to destabilise; memory is sized for weights plus a key-value cache rather than for optimiser state; and the workload is many small independent requests rather than one enormous synchronous job, so the interconnect requirements largely disappear. The constraint that replaces them is latency per request, which is a much harder thing to design for than throughput, because it cannot be recovered by batching more.

The memory ratio is the clearest way to see the split. Training in mixed precision holds roughly sixteen bytes per parameter once the optimiser's two moments and the fp32 master copy are counted, against about two for a served half-precision weight — so the state a training chip carries beyond the weights is several times the weights themselves. Everything else follows from that number: the memory-to-FLOPs ratio, the cost per chip, and why the same accelerator is rarely the right answer to both problems.