Training Corpus Contamination

Evaluation sets end up in training corpora because both are scraped from the same internet. The model then answers from memory and the benchmark reports understanding. Unlike Benchmark Contamination, which is about a benchmark's integrity, this is about what a crawl swept up before anyone looked.

Detection is a corpus-side problem: n-gram overlap against known evaluation sets, run at collection time, with the decontamination decision recorded. Doing it after the fact requires the corpus, which closed models do not publish — so for most reported numbers contamination is unfalsifiable rather than absent. Canary strings in benchmark files help only if the crawler and the trainer both respect them.

Flag a document when a long enough n-gram matches an evaluation item; the threshold trades false positives against missed paraphrase, and paraphrase evades any exact-match rule. Because the crawl predates the benchmark's existence in some cases and postdates it in others, publication date is a weak proxy that a temporal split can exploit — evaluating only on items created after the crawl cutoff.