Synthesis / Control
verifiedProsody Control
Prosody is the rhythm, stress and intonation that make speech mean something beyond its words. A synthesiser that gets phonemes right and prosody wrong sounds like a machine reading, and the reason is that the text underdetermines it — the same sentence has many correct readings and nothing in the writing says which.
Explicit control predicts duration, pitch and energy as separate quantities, which makes them editable and makes the model deterministic enough to debug. Implicit control samples a latent prosody variable, which sounds more natural and is harder to direct. The choice is between a system a user can steer and one that sounds better unattended, and it is not obvious which a product wants.
This is the multimodality Text to Speech describes, made concrete: p(prosody | text) has many modes, so predicting its mean gives the flat reading. Predicting duration, pitch and energy explicitly conditions the acoustic model on a chosen mode rather than averaging over them, which is why the flatness disappears when they are supplied.
explicit-control holds 50% of the budget; rest holds the remaining 50%.
Prosody supplied explicitly against prosody left to the model to choose, in equal units. Drag the explicit share up to watch the output become steerable — and stop surprising you in the ways that sounded natural.
Reviewed by opendroid · 2026-08-18
- arXiv:2304.09116 — NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers