Evaluation / Methods
verifiedStatistical Significance
Whether a difference between two numbers is bigger than the noise that produced them. Benchmark tables are full of comparisons where it has not been checked, and a great many reported improvements are inside the range two identical configurations would differ by anyway.
Two things are usually missing and each is enough on its own. A single run per configuration gives no estimate of variance, so there is nothing to compare the gap against — and re-running the same configuration with different seeds routinely moves the number by more than the claimed improvement. And a table comparing many methods on many datasets is making many comparisons at once, so some will look significant by chance unless that is corrected for.
The multiple-comparisons arithmetic is the part people find surprising. At a 5% threshold, twenty independent comparisons of things that do not differ produce a significant-looking result about once by construction — so a table with twenty cells contains an expected false positive before anyone has done anything wrong. That is why a paper reporting the best of many configurations has selected on the same noise it is measuring, and why the number to report is variance across seeds rather than a single best.
comparisons-made holds 17% of the budget; rest holds the remaining 83%.
Comparisons a results table makes at once, against the twenty a 5% threshold tolerates before one false positive is expected, in comparisons. Drag the comparison count up to watch it pass that mark — at twenty, one significant-looking result is what chance alone produces.
Reviewed by opendroid · 2026-08-18
- arXiv:1803.09578 — Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches
- arXiv:2103.03098 — Accounting for Variance in Machine Learning Benchmarks