Masked Language Modeling

Hide some of the words in a sentence and train the model to fill them back in. Because it can see both sides of a gap, it learns representations that account for what comes after — which a model predicting strictly left to right cannot. That bidirectionality is the whole trade: better representations, and no way to generate text one token at a time.

Mask around 15% of the tokens and predict them from the full bidirectional context. It produced the encoders that dominated classification and extraction for years, and it lost the generative frontier to causal models because the objective does not yield a sampler. The mask token also appears in training and never at inference, a mismatch the original work patched by replacing only some of the masks.

Maximise Σ sub _i∈M  log p(x sub _i  | x sub _visible ) over a mask set M, conditioning on both directions rather than on x sub _<t  alone. The signal is a fraction of the tokens per sequence against every token for a causal objective, so a masked model sees fewer predictions per unit of data and needs more passes over it to match.