Custom Kernel

Sometimes the compiler will not produce what you need and you write the kernel yourself. Languages like Triton put this within reach of someone who is not a CUDA specialist — you write in terms of blocks and the compiler handles the thread-level detail — which is why the number of hand-written kernels in open models rose rather than fell.

The case for writing one is a fusion the compiler cannot express or an algorithm it cannot derive, which is exactly what flash-attention is: not a faster matrix multiply but a different algorithm with the same output. The case against is maintenance — a custom kernel is pinned to assumptions about shapes and hardware that the surrounding code is free to change.

A block-level language exposes tiles as the unit of programming and infers the thread mapping, so the programmer chooses the memory schedule and the compiler chooses the parallel decomposition. That split is what makes the productivity gain real: the hard part of a fast kernel is data movement, and that is the part left with the human.