Speech / Representation
verifiedSpeech Representation Learning
Transcribed speech is expensive and raw audio is nearly free. Self-supervised speech models learn from the audio alone — hide parts of it, learn to identify what was hidden — and the representations that fall out turn an hour of labelled data into a usable recogniser. The pretext task is throwaway; what is kept is the encoder underneath it.
wav2vec 2.0 masks spans of latent frames and solves a contrastive problem against quantised targets from a learned codebook; HuBERT replaces the contrast with plain classification over cluster labels refined between rounds. Both need discrete targets because raw audio has no natural vocabulary, and the quantiser is what manufactures one. Masking spans rather than single frames is load-bearing: a one-frame gap is filled by interpolation, not understanding.
Mask a proportion of latent frames in spans of length M, then maximise the log-likelihood of the true quantised target q sub t against K distractors drawn from other masked positions: L = −log[ exp(sim(c sub t , q sub t )/κ) / Σ sub q̃ exp(sim(c sub t , q̃)/κ) ]. With start indices sampled at p ≈ 0.065 and M = 10, spans overlap enough that roughly half of all frames end up masked.
masked-frames holds 48% of the budget; rest holds the remaining 52%.
Latent frames hidden from the model against frames left visible, in frames. Drag the masked frames up to watch the pretext task get harder — too few and the neighbouring audio gives the answer away, and past about half there is little context left to answer from.
Reviewed by opendroid · 2026-08-18
- arXiv:2006.11477 — wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- arXiv:2106.07447 — HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units