Sequence Parallelism

Tensor parallelism splits the large matrix multiplies but leaves the normalization and dropout between them replicated, so every device still holds a complete copy of those activations. Sequence parallelism splits those regions along the sequence instead, and no device ever holds the whole thing. It costs no extra bandwidth: the all-reduce that was already happening is simply split into two cheaper collectives.

The regions between tensor-parallel blocks are computed per position — layer norm normalizes across features within one token, dropout is elementwise — so they are independent along the sequence and can be partitioned by token with no change to the result. Replacing each all-reduce with a reduce-scatter and an all-gather moves the same volume, which is why this is close to free. It removes much of the activation memory that made recomputation necessary in the first place.

Under tensor parallelism of degree t, the norm and dropout activations cost Θ(b·s·d) per device rather than Θ(b·s·d/t), because those regions are replicated rather than split. Partitioning them by sequence restores the divisor. Communication is unchanged: an all-reduce is a reduce-scatter followed by an all-gather, and the pair moves exactly what the whole did.