Pruning

Most weights in a trained network can be set to zero without hurting it. Pruning finds them and removes them — usually by magnitude, sometimes by a saliency estimate — and then fine-tunes to recover what was lost. Networks tolerate this to a degree that is still not fully explained.

Unstructured pruning gives the best accuracy per weight removed and no speedup at all on ordinary hardware, because a sparse matrix with arbitrary zeros is slower than a dense one. That mismatch is the entire practical story: the literature reports compression ratios and deployments need latency, and the two only meet under structured sparsity. Iterative prune-and-retrain beats one-shot pruning at the same final sparsity.

Rank weights by |w| (or by an estimate of ∂L/∂w·w), zero the smallest fraction, fine-tune, repeat. Magnitude works better than it should because training already drives unimportant weights toward zero — so the criterion is partly reading off a decision the optimiser already made.