Reward Shaping

Add extra rewards along the way so the agent gets feedback before the end. A robot rewarded only for reaching the goal learns nothing until it stumbles there by accident; rewarded a little for getting closer, it learns immediately. The obvious danger is equally immediate — reward getting closer and it may learn to hover just short of the goal forever.

There is a form that provably cannot do that, and it is worth knowing because it is the only guarantee in the area. Shaping built as the difference of a potential function between two states — a value assigned to each state, and the reward being the change in it — leaves the optimal policy unchanged, whatever potential you choose. Anything else is a bet that your hint is right, and Specification Gaming is what collects when it is not.

The reason potential-based shaping is safe is that its contribution telescopes: summed along any trajectory, the intermediate terms cancel and only the endpoints remain, so it adds a constant to every trajectory between the same two states and cannot reorder them. Nothing about which policy is best changes. Shaping that is not of this form does not telescope, so it can and does add more to some trajectories than others — which is exactly the ability to change the answer.