Systems / Distributed
verifiedPipeline Parallelism
Give each device a consecutive group of layers and pass activations along the chain. The problem is obvious the moment you draw the schedule: while the first stage works, every other stage has nothing to do. Splitting the batch into micro-batches keeps them all busy, and the more micro-batches are in flight the smaller the idle fraction — the part that remains is called the bubble.
Only the activations at stage boundaries cross the wire, which is far less than tensor parallelism moves, so this is the strategy that crosses machines. The costs are the bubble and the memory: every stage must hold activations for each micro-batch still in flight. Interleaved schedules shrink the bubble further and pay for it in communication.
With p stages and m micro-batches the bubble is (p−1)/(m+p−1) of the time: about 27% at p = 4 and m = 8, and under 5% at m = 64. It shrinks with m rather than with anything about the model, so the fix is always more micro-batches — bounded by the activation memory each one in flight demands.
bubble-steps holds 27% of the budget; rest holds the remaining 73%.
The pipeline bubble against the micro-batch steps that fill it, both as slots in the schedule. Drag the number of idle stages to watch the bubble take the run over.
Reviewed by opendroid · 2026-08-04
- arXiv:1811.06965 — GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
- arXiv:1806.03377 — PipeDream: Fast and Efficient Pipeline Parallel DNN Training
- arXiv:2104.04473 — Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM