Systems / Distributed
verifiedExpert Parallelism
A mixture-of-experts model holds more parameters than one accelerator can, so the experts are spread across devices. Each token then has to travel to whichever device owns the expert that was chosen for it, and come back. The routing decision stops being arithmetic and becomes network traffic.
Shard experts across devices and keep the rest of the model replicated. Each layer becomes two all-to-all collectives: one to dispatch tokens to the devices owning their chosen experts, one to return the outputs. Throughput is then bounded by interconnect bandwidth and by the slowest device, so a load-imbalanced router stalls the whole step. The capacity factor exists to bound this — tokens beyond an expert's capacity are dropped rather than allowed to skew the collective.
With E experts over D devices, each device holds E/D experts. For batch B and top-k routing, the expected tokens arriving at one device is B·k/D under uniform routing, and each device provisions capacity C = ⌈(B·k/D)·c⌉ for capacity factor c. Tokens beyond C are dropped, so drop rate rises with routing skew: for a device receiving a fraction p of tokens rather than 1/D, overflow is max(0, p·B·k − C).
6 tokens routed across 8 experts, sharded across 4 devices. Each token wakes its top 2. Experts 1, 8 stay asleep. 2 assignments were dropped where an expert exceeded its capacity of 2.
6 tokens routed across 8 experts, sharded across 4 devices. Each token wakes its top 2. Experts 1, 8 stay asleep. 2 assignments were dropped where an expert exceeded its capacity of 2.
Tokens dispatched across four devices, with dropped tokens marked where an expert exceeds capacity. Drag the capacity factor to trade wasted memory against dropped tokens.
Same primitive, engineer depth: how full each expert runs. An expert pinned at capacity is dropping tokens; one far below it is memory bought and not used. Drag the capacity factor to move every bar at once.
Dashed = dropped, expert at capacity (2)
Bar = share of each expert's capacity used
Reviewed by opendroid · 2026-08-04
- arXiv:2006.16668 — GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- arXiv:2101.03961 — Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity