Energy Cost of Inference

Training happens once; serving happens every time anyone asks a question. So the energy that matters for a deployed model is not what it cost to build but what it costs to run, and past enough traffic the second overtakes the first — often well within the model's serving life, and then keeps going.

Per request the number is small, which is what makes it easy to dismiss and easy to underestimate in aggregate. The levers are the ordinary serving ones seen from a different angle: batching amortises the weight reads that dominate a small request, quantisation moves fewer bytes for the same answer, and a smaller model serving the same traffic is the largest lever of all. Test-Time Compute runs directly against this — thinking longer is more energy per answer, every answer.

Cumulative inference energy is per-request energy times request count, so it is linear in traffic while training energy is a constant. Two lines with those shapes cross exactly once, and the crossing point is early for anything widely deployed: a model serving millions of requests a day passes its own training cost in a period measured in weeks or months rather than years. After the crossing the training figure is a rounding error, which is the opposite of how the two are usually discussed.