Adaptive Computation

Spend more on hard inputs than on easy ones. A fixed network does the same work for "what is 2+2" as for a research question, which is obviously wasteful — and the reason it persists is that variable work per input is awkward for hardware that likes doing the same thing to everything at once.

The distinction from Conditional Computation is which axis varies. That routes each token to a subset of experts and every token still costs the same; this varies the total work per input, so two requests in one batch finish at different times. That is the whole engineering difficulty — a batch runs at the pace of its slowest member, so saving compute on the easy inputs saves nothing at all unless the scheduler can retire them and refill the slot.

The gain is bounded by the share of inputs that are genuinely easy, and by how well easiness can be predicted before the work is done. Both terms matter and the second is usually the binding one: a perfect oracle over a workload that is 70% easy saves most of that 70%, and a predictor at chance saves nothing while adding its own cost. So the honest measure is the saving net of the predictor, on a real traffic mix rather than a benchmark's.