Compression-Quality Tradeoff

Every compression method has a point past which quality falls off a cliff rather than degrading gently, and the point is different for each. What makes this hard to reason about is that average benchmark scores hide it: a compressed model can match on aggregate while losing a specific capability entirely.

Perplexity is the worst offender — it moves by a fraction of a percent while multi-step reasoning collapses, because the rare tokens that carry a chain of thought contribute almost nothing to an average. Evaluating compression needs the capabilities you care about measured directly, and long-form and multi-step tasks degrade first. Compression methods also compose badly: quantised and pruned is worse than either at matched size.

Report quality against effective bits per parameter rather than against nominal compression ratio, since methods differ in what they count. The frontier is not convex in general — combining two methods can fall below the lower envelope of each — which is why a claimed ratio from one method does not transfer to a stack of them.