Systems
23 concepts, 23 reviewed and merged.
Distributed
- Collective CommunicationParallel training is mostly agreement: every worker computes something, and then they all have to end up holding the…verified
- Data ParallelismThe simplest way to use many accelerators is to put a full copy of the model on each, give each a different slice of…verified
- Expert ParallelismA mixture-of-experts model holds more parameters than one accelerator can, so the experts are spread across devices.verified
- Pipeline ParallelismGive each device a consecutive group of layers and pass activations along the chain.verified
- Sequence ParallelismTensor parallelism splits the large matrix multiplies but leaves the normalization and dropout between them…verified
- Tensor ParallelismWhen a single layer is too large for one device, split the layer.verified
- ZeROData parallelism keeps a complete copy of the optimizer state on every worker, which is the same numbers stored N…verified
Hardware
- Arithmetic IntensityEvery kernel does some arithmetic and moves some bytes.verified
- GPU Memory HierarchyThe memory closest to the arithmetic is tiny and enormously fast; the memory large enough to hold a model is far away…verified
- Kernel FusionA chain of simple operations — add a bias, apply an activation, scale the result — each reads its input from main…verified
Memory
Operations
- Determinism and ReproducibilityRun the same training script twice with the same seed and you will usually get two different models.verified
- Job SchedulingA cluster is shared, and the decision about which job runs next is where a lot of a research team's effective…verified
- ThroughputHow much work finishes per second, as opposed to how long any one piece takes.verified
- Training ObservabilityThe failure that costs the most is the one nobody notices.verified
Reliability
- Checkpointing StrategyA run across thousands of accelerators for six weeks will be interrupted many times, so the only question is how much…verified
- Elastic TrainingIf a job must have exactly 1,024 accelerators, losing one stops it.verified
- Failure RecoveryAt the scale large models are trained, hardware failure is not an incident — it is the weather.verified
- Loss SpikeA long run is going well and then the loss jumps — sometimes recovering on its own over a few thousand steps…verified
- StragglerIn a synchronous run every worker must finish its step before any of them can start the next, so the whole cluster…verified