Recommenders / Evaluation
verifiedOffline-Online Gap
A change that improves every offline metric can lose an A/B test, and this happens often enough that mature teams treat offline numbers as a filter rather than as evidence. The reason is that offline evaluation replays a world the new model would not have produced.
Three causes account for most of it: the candidate set was generated by the old system, the metric rewards agreeing with logged behaviour rather than being better than it, and position bias means a click depends on where something was shown as much as on what it was. Interleaving sits between offline and A/B and is more sensitive than either, and it is underused because it needs serving infrastructure.
Offline metrics estimate performance under the logging policy's distribution; the deployed model induces a different one, and the gap is a distribution shift rather than noise. Counterfactual estimators correct it only within the support of what was logged — an item never shown has no propensity, so no reweighting can say what would have happened.
in-support holds 25% of the budget; rest holds the remaining 75%.
Decisions the logs can speak to against decisions outside anything ever shown, in decisions. Drag the coverage up to watch offline evaluation become informative — outside this range, no correction recovers the answer.
Reviewed by opendroid · 2026-08-18
- arXiv:1907.06902 — Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
- arXiv:1902.10730 — Degenerate Feedback Loops in Recommender Systems