Feedback Loop

A recommender trains on what people clicked, and people click on what it recommended. The data is a product of the model, so any bias it has gets confirmed by the next round of training. Left alone this narrows what a system will ever show, and it happens without anyone choosing it.

This is the concrete case of the general problem Causal Inference describes — a model used to decide changes the distribution it is measured on. Logging propensities at serving time is what makes correction possible later, and it must be done then rather than reconstructed afterwards. A system that did not log them cannot be debiased, whatever method is applied.

Training on logged interactions estimates p(click | shown) while the quantity of interest is p(click | shown by an unbiased policy), and the two differ by the logging policy. Inverse propensity weighting corrects it if propensities were recorded — weighting each example by 1/p(shown) — at the cost of variance that grows as propensities approach zero, which is why clipping is standard.