Neural Audio Codec

A codec turns a waveform into a small number of symbols and back again. A neural one learns the symbols instead of hand-designing them, and reaches speech quality at a few kilobits per second that classical codecs need several times the budget for. The side effect mattered more than the compression: once audio is a sequence of discrete tokens, a language model can generate it.

An encoder downsamples the waveform, a residual vector quantiser turns each frame into a stack of codebook indices, and a decoder — usually adversarially trained — reconstructs. Residual is the trick: the first quantiser takes the signal and the second quantises what the first got wrong, and so on down the stack, so bitrate becomes a question of how many stages you keep at decode time rather than a separate model per rate.

Stage n quantises the residual r sub _n  = r sub _n−1  − e sub _n−1 (r sub _n−1 ) with r sub ₁  = z, so N stages over a codebook of size C cost N·log sub ₂ (C) bits per frame and the reconstruction is Σ sub _n  e sub _n (r sub _n ). The residual energy falls roughly geometrically in N, which is why the first stage carries most of the intelligibility and the later ones carry timbre.