Kernel Fusion

A chain of simple operations — add a bias, apply an activation, scale the result — each reads its input from main memory and writes its output back, even though the next step wants exactly what was just written. Fusing them into one kernel keeps the intermediate on chip and crosses the boundary once instead of three times. The arithmetic is unchanged; the traffic is not.

The gain is proportional to how memory-bound the operations were, so it is large for elementwise and normalization work and negligible for a big matrix multiply. Compilers fuse simple chains automatically; the cases worth writing by hand are the ones where fusion changes the algorithm rather than only the schedule — FlashAttention fuses the whole of attention and rewrites the softmax to make that legal.

k elementwise operations over n elements move 2kn values across the memory boundary unfused and 2n fused, so the traffic falls by a factor of k. The speedup approaches k only for work that was entirely bandwidth-bound and 1 for work that was not: the ceiling is the fraction of the original time that was traffic, which is Amdahl's law applied to bytes rather than to instructions.