RLHF

Instruction tuning teaches a model to imitate good answers. RLHF goes further: it trains the model against a score for how good an answer is, so it can be pushed toward responses better than anything in the demonstration data. Humans rank outputs, a reward model learns to predict those rankings, and the policy is optimised against it.

Three stages — supervised fine-tuning, reward model, then policy optimisation, classically with PPO. A KL penalty against the frozen reference model is not optional: without it the policy drifts into text that scores well and reads like nothing a person would write. It is the stage that made assistants usable, and it is expensive and finicky enough that simpler replacements are actively sought.

Maximise E[r sub _φ (x, y)] − β·KL(π sub _θ (y|x) ‖ π sub _ref (y|x)). The KL term is the whole design: β sets how far the policy may move from the model that produced coherent text, and the reward is only trustworthy near the distribution the reward model was trained on.