Audio Generation

Generating music or sound effects rather than speech. The problem is structurally different: there is no transcript to constrain it, coherence is needed over minutes rather than seconds, and quality is judged by whether it is pleasant rather than whether it is correct.

Hierarchical generation is the standard answer — a coarse structure over long timescales, filled in with detail — because a flat autoregressive model over audio tokens cannot hold minutes of context. Evaluation is the weak point: there is no equivalent of word error rate, so the field leans on distribution distances that correlate with human judgement loosely, and on listening tests that are expensive and rarely reported with error bars.

At a codec frame rate of 50 Hz and eight levels, a minute of audio is 24,000 tokens — beyond what a flat model attends over comfortably, which is why the hierarchy exists. Long-range structure is what listeners notice most and what token-level likelihood scores least, so the metric and the perception diverge exactly where the difficulty is.