Data Filtering

Most of a web crawl is not worth training on — boilerplate, spam, machine translation, pages that are mostly navigation. Filtering decides what survives, and the decision moves benchmark scores more than most architectural changes do. It is also the least published part of any frontier model.

The pipeline is a stack of cheap classifiers and heuristics: language identification, perplexity against a reference model, near-duplicate detection, quality classifiers trained on a curated positive set. Every filter is also a bias — a quality classifier trained on Wikipedia and books encodes what those look like, and removes dialects and domains that do not. Aggressive filtering can hurt: past a point it strips the diversity that made scale useful.

Each filter is a threshold on a score, and the aggregate keep rate is the product of the individual ones — so a stack of five filters each keeping 80% keeps a third. Because thresholds are usually tuned one filter at a time against a downstream metric, the interaction between them is rarely measured, and the stack's true selectivity is not what any single tuning run saw.