Offline RL

Learn a policy from a fixed dataset of past behaviour, with no chance to try anything and see what happens. That rules out the loop reinforcement learning normally depends on, and it introduces a specific trap: the algorithm will prefer actions the data never took, because their value was never corrected downward.

The problem is distributional shift and the fixes are all forms of pessimism — penalise values for actions outside the data, constrain the policy toward the behaviour policy, or model the return distribution and act conservatively. The tell that a method is under-constrained is a value estimate far above anything in the dataset, which is a claim about actions nobody ever tried. Evaluation is the hard part, since offline metrics have the same blind spot.

Q-learning's max over actions is evaluated at actions the dataset may never contain, so the bootstrap propagates errors that never get corrected — the loop that makes online RL self-repairing is exactly what is missing. Conservative methods add a penalty pushing down values on out-of-distribution actions, which trades a bias toward the data for the unbounded overestimation that would otherwise occur.