Word Embedding

A word becomes a vector, and words used in similar contexts end up near each other — not because anyone described their meanings, but because the vectors were fitted to predict the company each word keeps. That a purely distributional procedure recovers something that behaves like meaning is the surprising part, and it is the observation everything since rests on.

The distinction from Token Embedding is what the vector belongs to. This is a static table: one vector per word type, the same in every sentence, computed once. That is exactly its limitation — a word with two senses gets one vector that is the average of both, and no context can move it. Contextual embeddings, which is what a transformer produces at every layer, dissolve that limitation and are why the static kind became a teaching example rather than a component.

The famous arithmetic — that vector offsets encode relations — works because the training objective is a factorisation of a co-occurrence statistic, and offsets in the factorised space correspond to ratios of co-occurrence probabilities. It works partially and is oversold: analogy accuracy is much better on frequent, concrete relations than on anything else, and the standard evaluation excludes the query word from the answers, which flatters it more than most reports say.