Mixture of Depths

Let each token choose how many layers it passes through. Some tokens in a sequence carry most of the meaning and some are punctuation and filler, and processing both to full depth spends the same on each. A router at every block picks which tokens it will actually process, and the rest skip past it.

The design decision that makes this practical is a fixed capacity per block: exactly k of the n tokens are processed and the rest are routed around, so the compute per block is known in advance rather than data-dependent. That keeps the tensor shapes static, which is what lets it run efficiently at all — variable work per token would be the thing hardware is worst at, and this is the trick that avoids it.

With capacity k out of n tokens per block, the block does k/n of the work of a dense one, so a stack alternating dense and routed blocks at capacity one-eighth costs a little over half of dense. The saving is exact and known ahead of time, which is what separates this from every other adaptive method here — nothing depends on a prediction being right, only on the routing being sensible, and a bad router costs quality rather than the guarantee.