Autotuning

Nobody can predict which tile size, unroll factor and thread arrangement will be fastest, so the compiler tries many and measures. It is empirical optimisation over a configuration space, and it routinely beats hand-written kernels — not because the search is clever but because the space is larger than a person will explore.

The cost is compilation time, which can run to hours for a large model, so results are cached and shipped. A learned cost model replaces most of the measurement with prediction and is retrained per hardware target, which is the part that does not transfer. The honest framing is that autotuning converts engineer time into machine time, and the exchange rate is good.

Search a discrete configuration space by measuring candidates, guided by a cost model fitted to previous measurements. Because the objective is measured rather than differentiable, the methods are evolutionary or bandit-style, and the search's value is bounded by how much the best configuration beats a reasonable default — usually a factor of a few, occasionally an order of magnitude.