Evaluation / Validity
verifiedBenchmark Contamination
A model trained on most of the public internet has probably seen the test set. When that happens a benchmark stops measuring ability and starts measuring memorisation, and the score goes up either way. The problem is not that anyone cheated — it is that the training corpus is too large for anyone to know what is in it.
Exact-match decontamination is the usual defence and it is weak: a rephrased or translated test item survives n-gram filtering and still leaks the answer. Detection after the fact is harder than it sounds, since a model can score well on a contaminated set for legitimate reasons too. Held-out sets created after a model's cutoff are the only clean measurement, and they expire.
If a fraction c of a benchmark is memorised, observed accuracy is roughly c + (1−c)·a for true ability a, so contamination inflates most where the model is weakest. That is the worst possible bias: it compresses the gap between models exactly where the benchmark was supposed to discriminate.
contaminated holds 10% of the budget; rest holds the remaining 90%.
Share of a benchmark score attributable to memorised items against genuine ability. Drag the contamination rate to watch a weak model close the gap.
Reviewed by opendroid · 2026-08-04
- arXiv:2311.04850 — Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- arXiv:2403.07974 — LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code