Invariant Risk Minimization

If a relationship is causal it should hold in every environment; if it is a shortcut it will not. So collect data from several environments and ask for a representation on which the same predictor is optimal in all of them. What survives that filter is more likely to be mechanism than coincidence.

In practice it is empirical risk plus a penalty on how much the optimal predictor would change per environment. The results are contested — the penalty is hard to optimise, the linear analysis does not transfer cleanly to deep networks, and several papers report that careful ERM matches it on the benchmarks it was proposed for. The idea is right and the estimator is unsettled, which is worth saying plainly.

Minimise Σ sub _e  R super ^e (Φ) + λ·‖∇ sub _w|w=1  R super ^e (w·Φ)‖² over representations Φ, where the penalty measures how far a fixed classifier is from optimal in environment e. As λ grows the objective stops caring about average risk and starts demanding agreement across environments — the trade the whole method is.