Low-Resource Language

Most of the world's languages have almost no digital text. For them the usual recipe — scrape a corpus, train, evaluate on a benchmark — fails at every step, and the small amount of text that does exist is often religious material or machine translation output, which teaches a model the wrong register and sometimes its own errors.

Corpus quality is a bigger problem than corpus size here: automatically crawled low-resource corpora have been audited and found to contain large fractions of misidentified or non-linguistic content. Language identification itself fails on exactly the languages that need it most, so filters trained to keep good text discard the language entirely. Evaluation sets are frequently translations of English ones, which measures something narrower than it appears to.

Data volume follows a steep power law across languages, so the median language has orders of magnitude less text than the mean suggests. Under Data Scaling Laws a language below the repetition limit cannot reach a given loss by training longer — the constraint is unique tokens, and no schedule manufactures them.