Rate-Distortion

Lossless compression has a floor. Lossy compression has a curve: for every budget of bits there is a best achievable error, and for every error you will tolerate there is a smallest number of bits. Rate-distortion theory is that curve, and every codec — audio, image, or the quantiser inside a neural one — is an attempt to sit on it.

The practical lesson is that rate and distortion are not the whole story. A reconstruction can score well on squared error and sound or look wrong, because low distortion and high realism are different objectives that trade against each other — which is why neural codecs train against a discriminator rather than only a reconstruction loss, and why their outputs beat classical codecs at bitrates where the squared error is worse.

R(D) = min I(X;X̂) over reconstructions with E[d(X,X̂)] ≤ D — the fewest bits per symbol achieving distortion D. It is convex and decreasing, meeting the entropy at D = 0. Adding a perception constraint moves the whole curve: at fixed rate, requiring the output distribution to match the source's costs distortion, and that cost is provably non-zero.