Feature Attribution

Which parts of the input were responsible for this output? Attribution methods answer with one number per input feature, and they are the oldest and most used family in interpretability. They are also the most easily misread: a heatmap over pixels or tokens looks like an explanation whether or not it is one.

Gradient-times-input is the cheap version and is noisy. Integrated gradients accumulates gradients along a path from a baseline, which buys axioms — completeness and sensitivity — at the price of choosing that baseline, a modelling decision people routinely skip. Sanity checks matter here more than anywhere else: some published saliency maps survive randomising the model's own weights, which means they were showing edges in the image rather than anything the model did.

IG sub _i  = (x sub _i  − x′ sub _i )·∫ ∂F(x′ + α(x − x′))/∂x sub _i  dα over α from 0 to 1, approximated by a Riemann sum of 20 to 300 steps. Completeness says the attributions sum to F(x) − F(x′), which is exactly why the baseline x′ is not a detail: it decides what the explanation is relative to.