Reward Model

Nobody can write down a loss function for 'a helpful answer'. But people can reliably say which of two answers is better, and that comparison is cheap. A reward model is trained on those comparisons until it can score a response on its own — turning a judgement nobody can specify into a number an optimiser can use.

Usually the policy model with the language head replaced by a scalar output, trained on pairs labelled by which response a human preferred. It is the component that decides what gets optimised, and every flaw in it becomes a flaw in the final model — a reward model that likes long answers produces a model that writes long answers.

Under the Bradley–Terry model, P(y sub _w  ≻ y sub _l  | x) = σ(r(x, y sub _w ) − r(x, y sub _l )), so the loss is −log σ(r(x, y sub _w ) − r(x, y sub _l )) over preference pairs. Only differences are identified: adding a constant to r changes nothing, which is why the absolute score is not meaningful and only comparisons are.