DPO

RLHF trains a reward model and then optimises against it with reinforcement learning — two models, two stages, and a great deal that can go wrong. DPO shows the reward model was never needed: the same optimum can be reached by a supervised loss on the preference pairs themselves. One stage, no sampling, no value network.

Train directly on preference pairs with a classification-style loss, keeping a frozen reference model for the ratio. Far simpler and cheaper than PPO and widely adopted for that reason. It is not free: it optimises the preferences it was given rather than exploring, so it cannot discover a response better than the ones it was shown.

The RLHF optimum is π*(y|x) ∝ π sub _ref (y|x)·exp(r(x,y)/β), which inverts to r(x,y) = β·log(π*(y|x)/π sub _ref (y|x)) + β·log Z(x). That last term depends on the prompt alone, so it cancels between two responses to the same prompt — which is what makes the substitution possible rather than merely tidy. Substituting into the Bradley–Terry loss removes r entirely, leaving −log σ(β·log(π sub _θ (y sub _w )/π sub _ref (y sub _w )) − β·log(π sub _θ (y sub _l )/π sub _ref (y sub _l ))) — the reward model was implicit in the policy all along.