Memory Bandwidth Wall

Every accelerator generation adds far more arithmetic throughput than memory bandwidth. The gap has widened for decades, and the effect is that operations which used to be limited by how fast the chip could compute are now limited by how fast it can be fed. Buying a faster chip stops helping, which is a strange and specific kind of disappointment.

This is why the same kernel can be compute-bound on one generation and memory-bound on the next without a line of it changing. It is also why the optimisation that pays has shifted from reducing arithmetic to reducing data movement — fusing kernels so intermediates never reach memory, recomputing values rather than storing them, and choosing algorithms with better reuse even when they do more arithmetic. Arithmetic Intensity is the per-kernel measurement; this is the trend that keeps moving the threshold it is measured against.

The crossing point sits where a kernel's arithmetic per byte equals the machine's FLOPs per byte of bandwidth, and that machine ratio has climbed steadily — an accelerator demanding around ten operations per byte a decade ago now demands hundreds. So a kernel with fixed reuse does not change while the bar it must clear rises underneath it, and the share of a workload that is memory-bound therefore grows with every generation without anyone rewriting anything.