PPO

Policy gradient methods improve a policy by pushing up the probability of actions that scored well. Push too far on one batch and the policy moves somewhere its own estimates no longer describe, and the run collapses. PPO clips each update so the new policy cannot stray far from the old one, which buys stability at the cost of taking smaller steps.

Optimise a surrogate objective in which the probability ratio between new and old policy is clipped to a narrow band, so an update that would move a token's probability drastically gets no additional credit for doing so. It is the workhorse of RLHF despite being general-purpose RL, and it brings a value network of its own alongside the policy. The four-model memory bill usually quoted — policy, value, reference and reward — is RLHF's rather than PPO's: the last two belong to the pipeline, not the algorithm.

Maximise E[min(r sub _t   sub _t , clip(r sub _t , 1−ε, 1+ε)  sub _t )] with r sub _t  = π sub _θ (a sub _t |s sub _t )/π sub _old (a sub _t |s sub _t ) and ε typically 0.2. The min makes the bound pessimistic: clipping only removes incentive to move further, it never rewards it, so the objective is a lower bound on the unclipped one.