Tiling

A matrix multiply that streams straight through memory is slow whatever the arithmetic peak says. Tiling breaks the computation into blocks sized to fit in fast memory, so each value loaded is used many times before it is evicted. Every fast kernel in this graph is tiled, including the one FlashAttention is named for.

Tile sizes are chosen against a specific memory hierarchy — register file, shared memory, L2 — so a kernel tuned for one accelerator is mistuned for the next. This is the concrete reason kernels are re-tuned per architecture rather than ported, and the reason autotuning exists rather than a table of good sizes.

With a tile of size T, each loaded element is reused about T times, so arithmetic intensity rises roughly linearly in T until the tile stops fitting in the level being targeted. The optimum is the largest tile that fits, which makes the choice a discrete search over a small set of shapes constrained by capacity rather than a continuous optimisation.