Streaming ASR

A transcript that arrives after you stop talking is a different product from one that appears as you speak. Streaming recognition emits words while the audio is still arriving, which means committing to a word before hearing what follows it. Every bit of future context that makes the guess better also makes the caption later, and past roughly 300 ms a listener stops reading and starts waiting.

The constraint is architectural rather than an optimisation: an encoder that attends over the whole utterance cannot run until the utterance ends. Streaming models bound how far ahead each frame may see — chunked attention, a fixed right context, or none at all — and pay for it in accuracy. RNN-T is the common choice because its output depends on previous outputs without depending on future inputs, which is the one dependency streaming can afford.

With right context R, frame i attends over frames j ≤ i + R, so the output at i waits for R further frames of audio: a latency of R·H/sr seconds at hop H. Chunked attention amortises that — a chunk of C frames shares one boundary, giving mean latency about (C/2 + R)·H/sr — which is why chunking beats per-frame lookahead at equal accuracy.