Foundations / Representation
verifiedByte-Pair Encoding
Byte-pair encoding builds a vocabulary by repetition. Start with single characters, count which adjacent pair occurs most often, merge it into one new symbol, and repeat. Common words fuse into single tokens; rare ones stay in pieces. The vocabulary is discovered from the corpus rather than chosen by anyone.
Run the merge loop until the vocabulary reaches its target size, then store the ordered merge list — applying it in the same order is what makes encoding deterministic. Operating on bytes rather than characters removes the unknown-token case entirely, since every input is representable. The frequency statistics come from the training corpus, so its composition is baked into the tokenizer.
Given a corpus of symbol sequences, repeatedly select the pair (a,b) maximising its count and replace all occurrences with a new symbol ab, for k iterations. The result is a vocabulary of size |Σ| + k and an ordered merge table applied greedily at encode time.
8 values. The left group decays steeply; the right group is 72% of the way to flat, and reads flatter than the left.
Token frequencies before merging and after, ordered most common first. Drag the merge count to watch frequent pairs fuse and the distribution redistribute.
Reviewed by opendroid · 2026-08-04
- arXiv:1508.07909 — Neural Machine Translation of Rare Words with Subword Units
- arXiv:1909.03341 — Neural Machine Translation with Byte-Level Subwords