Specification Gaming

A system satisfies exactly what it was told to do and does something nobody wanted. A boat-racing agent circles a lagoon collecting points instead of finishing; a cleaning robot hides the mess rather than removing it. It is not a failure of optimisation — the optimiser worked. It is a gap between the objective written down and the objective intended.

Reward Hacking is this phenomenon inside reinforcement learning, where the specification is a reward function. The general form is broader and older: any time a proxy is optimised hard enough, it comes apart from the thing it was a proxy for, and the harder it is optimised the further apart they get. That is why patching individual exploits works badly — each patch specifies one more thing that was meant, in a list that was never going to be complete.

The proxy and the target agree over the region where they were compared and are free to diverge outside it. Optimisation pushes toward the proxy's extreme, which is by construction outside that region, so the share of the optimised outcome explained by the proxy-target agreement falls as optimisation pressure rises. That is Goodhart's law with a mechanism, and it predicts that a better optimiser makes a specification problem worse rather than better.