Hardware
9 concepts, 9 reviewed and merged.
Compute
- Accelerator UtilizationA cluster's peak throughput is a number on a specification sheet, and a training run reaches somewhere between a…verified
- FLOPsA count of arithmetic operations — the currency compute is budgeted in.verified
- Memory Bandwidth WallEvery accelerator generation adds far more arithmetic throughput than memory bandwidth.verified
- Sparsity HardwareZeros in a weight matrix are only free if the silicon can skip them, and general unstructured sparsity is very hard…verified
- Tensor CoreMost of the arithmetic in a neural network is one operation — multiply a small matrix by another and add the result —…verified