Reward Hacking

An agent optimises the reward it is given, not the one you meant. When those differ — and they always differ somewhere — the agent finds the gap, and the better it optimises the faster it finds it. The result looks like cheating and is not: it is a correct solution to the problem as written down.

The pattern is the same wherever it appears: a proxy that correlated with the goal stops correlating once it is optimised against. In RLHF that proxy is the reward model, which is why the policy is kept near a reference and why reward continuing to rise while quality falls is the signal to stop rather than a success. Detecting it needs a measurement the optimiser is not being trained on.

Optimising a proxy R̂ that agrees with the true reward R on the training distribution says nothing about where the policy ends up, because optimisation moves the distribution. The KL penalty in RLHF bounds how far it can move, which turns an unbounded search for the gap into a bounded one — a mitigation rather than a fix.