Tensor Core

Most of the arithmetic in a neural network is one operation — multiply a small matrix by another and add the result — so accelerators grew dedicated units that do only that, hundreds of times faster than the general-purpose lanes beside them. Everything else the model does still runs on the slow lanes, and that division is what modern kernel engineering is organised around.

The consequence is that a kernel's speed depends less on how many operations it does than on which units they land on. A softmax, a layer norm and an activation are all elementwise, all excluded from the fast path, and all cheap in FLOPs — so a naive implementation can spend most of its wall clock on the operations that account for a rounding error of the arithmetic. Fusing them into the matmul kernel, so the intermediate never leaves the chip, is the standard fix and the reason kernels are written by hand at all.

Matmul FLOPs grow with the square of the layer width while elementwise FLOPs grow only linearly with it, so the fraction the fast units can take rises as models get wider — which is why the same accelerator delivers a much larger share of its peak on a large model than a small one. The remainder is the ceiling: if a tenth of the time is elementwise, no amount of matmul acceleration takes the kernel below that tenth, which is Amdahl's law wearing a hardware badge.