Mixture-of-Experts

A dense model pushes every token through every parameter it owns. A mixture-of-experts model keeps many parallel sub-networks and wakes only a couple per token, chosen by a small router. The model can hold far more knowledge than it spends compute on, because you pay for the experts you wake up, not the ones you own.

Replace the feed-forward block with N expert FFNs and a learned router. The router scores every token against all N experts and dispatches it to the top-k, usually k=2. Compute scales with k, not N, so parameter count and FLOPs decouple. The cost moves from compute to memory and to balance: every expert must be resident, and a router that favours a few experts wastes the rest while overflowing its favourites past their capacity factor.

For token x, the router computes g = softmax(W sub _r  · x) over N experts and selects T = top-k(g). The output is y = Σ sub _i∈T  (g sub _i  / Σ sub _j∈T  g sub _j ) · E sub _i (x), so the mixture stays a convex combination of the chosen experts. Training adds an auxiliary load-balancing loss, typically α · N · Σ sub _i  f sub _i  · P sub _i , where f sub _i  is the fraction of tokens routed to expert i and P sub _i  the mean router probability for it. The product is minimised when both are uniform.