Interconnect Topology

Accelerators in a large cluster are not equally close to each other. Eight in the same server share a fast dedicated fabric; two in different racks talk over a network an order of magnitude slower. Which accelerators hold which part of the model therefore decides how much of the traffic gets the fast path, and a placement that ignores this can leave most of the machine waiting.

The standard arrangement follows the hierarchy rather than fighting it. Tensor parallelism, which exchanges activations at every layer and is the most communication-hungry, is kept inside a single node; pipeline parallelism, which sends one activation per stage boundary, is allowed to cross nodes; data parallelism's gradient all-reduce sits between them and is where topology-aware collectives earn their place. Getting the mapping backwards is one of the few configuration mistakes that costs a factor rather than a percentage.

The cost of a placement is the share of traffic that has to cross the slow tier. Keeping the chattiest parallelism inside a node holds that share near zero; spreading it across nodes sends the same volume over links with a fraction of the bandwidth, and because the collective is synchronous, the slow tier sets the pace for everyone. The ratio between the tiers is what turns a placement decision into a throughput decision — the volume did not change, only which wire carried it.