Structured Sparsity

Zeros only help if the hardware can skip them. Removing whole channels, heads or blocks produces a smaller dense model that runs faster on anything; removing scattered individual weights produces a model that is smaller on paper and no faster in practice. Structure is what converts compression into latency.

The cost is accuracy per parameter removed: a structured constraint throws away useful weights alongside useless ones because they share a group. Hardware meets it partway — 2:4 sparsity, where two of every four adjacent weights are zero, is supported directly by recent tensor cores, and it is a compromise between arbitrary sparsity and whole-channel removal rather than a solution to either.

Impose the sparsity pattern as a constraint on groups rather than on individual weights: minimise L subject to whole rows, channels or 2:4 blocks being zero. The achievable accuracy at a given zero fraction is strictly worse than unstructured, and the achievable speedup is strictly better — the two curves cross wherever the deployment's bottleneck actually is.