Foundations / Representation
verifiedTokenization
A model does not see text, it sees numbers. Tokenization is the step that chops a string into pieces and assigns each an id. The choice of pieces is a trade: characters are universal but make sequences long, whole words are compact but leave the model helpless on anything unseen.
Subword schemes take the middle: frequent words stay whole, rare ones split into parts that were seen in training. Vocabulary size trades against sequence length, and both cost money — the embedding table scales with the vocabulary while attention scales with the square of the sequence. Tokenizer choices are effectively permanent, since changing them invalidates every trained weight tied to the table.
A tokenizer is a map from strings to sequences over a vocabulary V. Embedding parameters scale as |V|·d sub model , while attention cost scales as n². For a fixed corpus, larger |V| lowers mean n, so the total is minimised somewhere in between rather than at either end.
8 values. The left group decays steeply; the right group is 43% of the way to flat, and reads flatter than the left.
How the same text divides into tokens at one vocabulary size and another. Drag the vocabulary up to watch sequences shorten as the table grows.
Reviewed by opendroid · 2026-08-04
- arXiv:1508.07909 — Neural Machine Translation of Rare Words with Subword Units
- arXiv:1808.06226 — SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing