Script and Orthography

Writing systems differ in ways that reach all the way down to the byte level. Some scripts have no spaces between words, some encode the same character in several ways, some render differently depending on neighbours. A pipeline built for Latin script quietly mishandles all of it, usually without erroring.

Unicode normalisation is the first thing to get right and the easiest to skip: the same visible string can have multiple byte representations, so deduplication misses duplicates and lookups miss matches. Word-boundary assumptions are the second — a tokenizer that pre-splits on whitespace has already decided that scripts without spaces are one long word, which is why byte-level tokenizers handle them better without anyone intending it.

Byte-level encoding sidesteps the boundary question at the cost of sequence length, which is exactly the fertility cost Tokenizer Fairness measures. Normalisation is a many-to-one map, so skipping it inflates apparent vocabulary and splits statistics that should have been pooled — an error that shows up as slightly worse everything rather than as a failure.