Foundations / Statistics
verifiedHidden Markov Model
A sequence of states you cannot see, each producing an observation you can. Speech is the canonical case: the phonemes are hidden, the audio is what arrives, and the task is to recover the first from the second. It was the backbone of speech recognition for decades and it is the ancestor of everything sequential in this corpus.
The Markov assumption — the next state depends only on the current one — is what makes it tractable and what limits it. Dynamic programming then gives exact answers to all three questions you want: how likely is this observation sequence, what is the most likely state path, and what parameters best explain the data. Neural sequence models gave up the exactness and the interpretability in exchange for dropping the assumption, and that is the trade the whole lineage represents.
The state path has exponentially many possibilities and the forward and Viterbi recursions evaluate them in time linear in the sequence and quadratic in the state count, by reusing shared prefixes. That is the same structural insight CTC uses to sum over alignments, arrived at in the 1960s, four decades before CTC — which is why CTC feels familiar to anyone who learned this first, and arbitrary to anyone who did not.
10 queries against 10 keys; a brighter cell means more of that query's attention went to that key. Each query sees up to 1 positions ahead of itself; the rest is masked.
States against the states they may follow from, under the Markov assumption. Drag the lookahead up to watch the model see further back — at one, the whole of history is the current state, which is the assumption that makes the recursions exact and the model limited.
Reviewed by opendroid · 2026-08-18
- arXiv:1206.6426 — A fast and simple algorithm for training neural probabilistic language models
- arXiv:1301.3781 — Efficient Estimation of Word Representations in Vector Space
Origin · not linkable
- Rabiner 1989 — A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition · Proceedings of the IEEE 77(2) · doi:10.1109/5.18626