Data Parallelism

The simplest way to use many accelerators is to put a full copy of the model on each, give each a different slice of the batch, and average the gradients before stepping. Every copy stays identical, so the result is one large-batch update computed in parallel.

An all-reduce of the gradients every step. It scales until the communication stops overlapping with computation, or until the effective batch passes the point where more examples stop helping. It also requires the whole model and its optimizer state to fit on one device, which is the constraint sharding exists to remove.

Each of N workers computes ∇L sub _B sub i   over |B|/N examples; an all-reduce produces the mean. Communication per step is Θ(P) for P parameters regardless of N with ring all-reduce, so the cost is bandwidth-bound rather than worker-count-bound.