Sparsity Hardware

Zeros in a weight matrix are only free if the silicon can skip them, and general unstructured sparsity is very hard to skip — the hardware cannot know in advance which lanes to idle. So accelerators support one narrow, rigid pattern instead: in every group of four consecutive weights, exactly two must be zero. Fit that pattern and the matmul runs at twice the rate; miss it by one weight and you get nothing.

The rigidity is the design, not a limitation someone forgot to fix. A fixed two-of-four pattern means the index of every surviving weight fits in two bits and the selection multiplexer is the same width for every group, so the decode costs almost no area. Anything more flexible needs per-row metadata and variable-width gather, which is where the area and the power go. Training to the pattern is a pruning-and-retraining recipe rather than something a trained model happens to satisfy.

The ceiling is exactly two, and it is worth seeing why nothing gets past it: the pattern fixes the skipped fraction at one half by definition, so the arithmetic saving cannot exceed a factor of two however sparse the underlying model really is. A model that is 90% zeros still runs at 2× under this scheme, because the hardware skips two of every four and no more. That is the whole trade — a guaranteed, modest, universally-available speedup instead of a large and unreliable one.