FLOPs

A count of arithmetic operations — the currency compute is budgeted in. Scaling laws are stated in it, training runs are costed in it, and it is deliberately a count rather than a duration, because a duration depends on the machine and a count does not.

The gap between the count and the wall clock is the thing to watch, and it is large. A model's FLOP count is fixed by its shape; the fraction of the machine's peak actually delivered is not, and large training runs land somewhere between a third and two thirds. So the count says what a job costs in principle and utilisation says what it costs in practice, and quoting either alone hides the other. Note also the two senses, constantly conflated: FLOPs the count of operations, and FLOP/s the rate a machine sustains.

For a transformer the forward pass costs roughly 2 FLOPs per parameter per token — a multiply and an add — and the backward pass roughly twice that, so training costs about 6ND for N parameters and D tokens. That approximation is what makes a compute budget tractable at all, and it is where the compute-optimal result comes from: fix 6ND and ask how to split it between N and D, and the answer is an interior optimum rather than an endpoint, which is why neither the largest model nor the most data wins.