N-Gram Model

Predict the next word by counting how often it followed the previous few in a large pile of text. No parameters to learn beyond the counts, no training in any modern sense. It was the language model for forty years, it is the baseline every neural model was argued against, and Perplexity exists because of it.

The whole engineering problem is what to do about the counts you did not see. A four-word sequence absent from the corpus gets probability zero and takes the whole sentence to zero with it, so every usable system backs off to shorter contexts and redistributes mass from what was seen to what was not. The smoothing scheme mattered enormously, and the best of them held up against early neural models for longer than the field likes to remember.

The count table grows as the vocabulary to the power of n, so going from three words of context to five multiplies the space by the vocabulary squared while the corpus stays the same size — the fraction of possible n-grams ever observed collapses toward zero. That is the sparsity wall, and it is the reason context length was capped at four or five words by arithmetic rather than by choice. Learning a dense representation instead of counting is precisely what removed it.