Spectrogram

A waveform is tens of thousands of numbers a second and almost none of them mean anything alone. Chop the signal into short overlapping windows, ask each window which frequencies it holds, and stack the answers left to right. Time runs across, pitch runs up, and speech becomes something you can look at. Nearly every speech system starts here rather than at the raw samples.

The short-time Fourier transform: window the signal — 25 ms of Hann is the usual — take the FFT, keep the magnitudes, hop forward 10 ms and repeat. Window length is the one real decision and it is a trade nobody escapes: a long window resolves frequency and smears time, a short one does the reverse. Most pipelines then warp the frequency axis to the mel scale and take a log, which spends resolution where hearing is sharp and compresses the rest.

A window of N samples at rate sr gives N/2 + 1 bins spaced Δf = sr/N apart, one frame every H samples. Frequency and time resolution are reciprocal — Δf · Δt = 1 — so halving one doubles the other, always. The magnitudes |X sub _t,f | are kept and the phase is thrown away, which is why turning a spectrogram back into audio needs Griffin–Lim or a learned vocoder rather than an inverse FFT.