Compilation Cache

Compiling and autotuning a large model can take longer than running it. A cache keyed on the graph, the shapes and the hardware makes that a first-run cost rather than a per-run one — and getting the key wrong is worse than having no cache, because a stale hit produces a kernel compiled for something else.

The key has to include everything the generated code depends on: operator versions, hardware model, driver, compiler flags, and the shapes. Cache misses in production usually trace to one of those changing invisibly — a driver update, a different accelerator model in the same fleet — which is why cold-start latency is reported as a distribution rather than a number.

Amortised cost is compile time divided by the number of runs hitting the entry, so the cache pays off in proportion to shape stability. A serving fleet with many shapes and short-lived processes may never amortise it, which is the case bucketing exists to create rather than merely to improve.