Ablation

Remove one piece and measure what happens. It is how nearly every claim in the field is supported — this component is why the number improved — and it is the closest thing to a controlled experiment available when the system is too complex to reason about analytically.

The common failure is that removing a component changes more than the component. Delete a regularisation term and the best learning rate moves, so what is measured is the term plus a now-mistuned optimiser. An honest ablation re-tunes what the removal invalidated, which is expensive and is the reason most published ones do not. The second failure is coverage: removing pieces one at a time cannot see an interaction, and interactions are common.

One-at-a-time removal explores a number of configurations linear in the component count while the space of subsets is exponential, so the fraction of the space examined collapses as a system gains parts. That is fine when effects are additive and misleading when they are not — and a set of individually-small ablation results that do not add up to the headline gain is the signature of an interaction the design could not see.