Shape Specialization

A kernel compiled for a batch of thirty-two is not the kernel you want for a batch of one. Compilers specialise on shapes to generate good code, and every distinct shape then triggers a recompile — which is fine in training, where shapes repeat, and painful in serving, where sequence lengths vary with every request.

The two answers are dynamic shapes, where a dimension is compiled symbolically at some cost in quality, and bucketing, where inputs are padded up to a small set of sizes so the cache always hits. Bucketing wastes compute on padding and is usually the better trade, which is a useful example of a crude fix beating an elegant one on measurements.

With b distinct shapes the compiler holds b variants, and each cold shape costs a full compilation before the first token. Padding to k buckets bounds that at k variants while wasting the difference between the true and padded length — so the choice is compile-time variance against a fixed fraction of throughput.