Speech Recognition

Turning audio into text used to mean four separate models — acoustics, pronunciation, language, and a decoder to reconcile them. Now it is usually one network that reads a spectrogram and writes characters. What makes it hard is not the size of the vocabulary but the variation: the same sentence differs by speaker, room, microphone, and mood, and none of that is in the transcript you train against.

Two families dominate. CTC-style models emit one distribution per frame and need no alignment; attention encoder-decoders read the whole encoded utterance and emit tokens autoregressively, buying an implicit language model at the cost of streaming. Whisper's result was that neither architecture was the bottleneck — supervision was, and 680k hours of weakly-labelled audio bought a robustness that careful in-domain training had never reached.

Decode ŷ = argmax sub _y  p(y|x) over token sequences given acoustic frames x. The two lengths are wildly unequal — |x| runs to a hundred frames per second of audio while |y| is a handful of tokens — so every architecture here is in some form a length reduction: striding convolutions, frame stacking, or attention that reads many frames per output step.