Tokenizer Fairness

A tokenizer trained mostly on English gives English the shortest encodings. The same sentence in another language can cost several times more tokens — which means more money per request, more of the context window consumed, and worse quality at the same budget. It is a fairness problem that arrives disguised as an implementation detail.

The effect is largest for scripts poorly covered in the training corpus, where words fragment into bytes. Because API pricing is per token, speakers of those languages pay more for the same content, and because the context window is in tokens, they get less of it. Rebalancing the tokenizer's training corpus is cheap relative to retraining a model and is rarely done, which is the part worth being direct about.

Fertility — tokens per word — measures it directly, and it ranges from near one for English to several for morphologically rich or under-covered languages. A fertility of three means three times the cost, a third of the effective context, and an effectively shorter memory, all from a preprocessing choice made once before training.