Synthesis / Representation
verifiedSpeech Tokenization
Once Neural Audio Codec turns audio into discrete tokens, speech becomes a sequence-modelling problem and everything built for language applies to it. The tokens split into two kinds that do different jobs: semantic tokens carrying what was said, acoustic tokens carrying how it sounded.
That split is the design most audio language models are built on. Semantic tokens are predicted first because they are what the text determines; acoustic tokens are filled in afterwards, conditioned on a short prompt that fixes the voice. Generating acoustic tokens autoregressively across all codebook levels is prohibitively long, which is why parallel and coarse-to-fine schemes exist rather than being optimisations.
A codec at N residual levels and a frame rate f produces N·f tokens per second — thousands per second at useful quality, against tens for text. That ratio is the whole engineering problem: sequence length is the binding constraint, so every method either lowers the frame rate, predicts levels in parallel, or gives up autoregression.
acoustic-tokens holds 75% of the budget; rest holds the remaining 25%.
Acoustic tokens describing how it sounded against semantic tokens describing what was said, in tokens. Drag the acoustic levels up to watch them swamp the sequence — they already outnumber the semantic ones at any useful quality, which is why they are rarely generated one at a time.
Reviewed by opendroid · 2026-08-18
- arXiv:2209.03143 — AudioLM: a Language Modeling Approach to Audio Generation
- arXiv:2305.09636 — SoundStorm: Efficient Parallel Audio Generation