Token Embedding

A token id is just a number with no meaning — id 400 is not twice id 200. The embedding table gives each id a vector instead, learned during training, so that tokens used in similar ways end up pointing in similar directions. It is the step that turns arbitrary labels into geometry the model can do arithmetic on.

A lookup table of shape |V| × d sub _model , indexed by token id. It is often the single largest parameter tensor in a small model, and is frequently tied to the output projection so the same matrix maps in and out. Tying saves |V|·d sub _model  parameters and usually costs nothing in quality.

E ∈ ℝ super ^{|V|×d sub _model}  , with the embedding of token t being row E sub _t . With weight tying, the output logits are h·Eᵀ rather than h·W for a separate W. The transformer's original scheme also scales embeddings by √d sub _model  before adding positional encoding.