Multilingual / Data
verifiedLanguage Imbalance
Web text is overwhelmingly a handful of languages. A corpus sampled proportionally gives some languages billions of tokens and others a rounding error, and a model trained that way is a strong English model with a multilingual veneer. Every multilingual corpus therefore resamples, and how it resamples is a decision with consequences.
Temperature sampling is the standard lever: raise the temperature and low-resource languages are oversampled relative to their share. Too little and they are ignored; too much and their small corpora are repeated many times, which is memorisation rather than learning. The right setting depends on how much unique text each language actually has, not on its proportion — which is the quantity nobody reports.
Sample language i with probability proportional to p sub i super 1/T . At T = 1 this is the natural distribution; as T grows it approaches uniform. The epochs a low-resource language sees rise with T, so the setting is bounded by the repetition limit Data Scaling Laws describes — roughly four epochs before repeated tokens stop paying.
16 values. The left group decays steeply; the right group is 36% of the way to flat, and reads flatter than the left.
Training mass across languages, as the web supplies it and as resampling evens it. Drag the temperature up to watch the distribution flatten toward uniform — and the smallest corpora start repeating.
Reviewed by opendroid · 2026-08-18
- arXiv:1911.02116 — Unsupervised Cross-lingual Representation Learning at Scale
- arXiv:2004.09095 — The State and Fate of Linguistic Diversity and Inclusion in the NLP World