Activation Patching

To find out whether a component causes a behaviour, replace its activations with the ones it had on a different input and see whether the answer changes. It is the closest thing interpretability has to an experiment: not "this correlates with the output" but "change this and the output changes".

Run a clean prompt and a corrupted one, cache both, splice one component's activations from the first into the second, and measure how much of the logit difference comes back. Sweeping that over every layer and position is what produces the map that localises a behaviour. The failure to keep in view is that patching finds components sufficient to restore a behaviour, which is not the same as the components the model uses to produce it.

With clean run c and corrupted run c′, patch component h and report (D(c′ with h←c) − D(c′)) / (D(c) − D(c′)), where D is the logit difference between the right answer and a distractor. One is full recovery and zero is none; the denominator is what makes the number comparable across prompts of different difficulty.