Speech / Synthesis
verifiedText to Speech
Reading a sentence aloud is not one problem but two: deciding how it should sound, and producing a waveform that sounds that way. Neural vocoders largely settled the second. The first is harder than it looks, because a sentence has many correct readings, and a model trained to minimise error will average them into a reading nobody would give.
The usual pipeline predicts a mel spectrogram from text and hands it to a vocoder that produces samples. The failure mode of the first stage is oversmoothing: trained under an L1 or L2 loss the model predicts the conditional mean over every plausible prosody, and the mean of many sharp spectra is a blurred one that sounds muffled. Adversarial losses, flows, diffusion, and explicit duration and pitch prediction are all ways of not being asked for the mean.
Under squared error the optimal prediction is E[y|x], and prosody makes p(y|x) genuinely multimodal, so that expectation lands between the modes rather than on any of them. Adversarial and likelihood-based objectives instead reward samples lying on the data manifold, which is why they recover detail that a loss seeking the mean cannot.
16 values. The left group decays steeply; the right group is 36% of the way to flat, and reads flatter than the left.
Harmonic detail in one frame, as recorded and as a mean-seeking loss predicts it. Drag the regression weight up to watch the peaks collapse toward their own average — the mean of every plausible reading is a reading that sounds muffled.
Reviewed by opendroid · 2026-08-18
- arXiv:1712.05884 — Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
- arXiv:2010.05646 — HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis