Policy Gradient

Rather than learning what each state is worth and acting greedily, adjust the behaviour itself: make the actions that led to reward more likely and the others less. It works where the action space is continuous or enormous, and it optimises the thing you actually care about rather than a proxy for it.

The estimator is high variance because a whole trajectory's reward is credited to every action in it, so nearly all the engineering is variance reduction — subtract a baseline, use an advantage rather than a raw return, and clip or constrain the step so one bad batch cannot destroy the policy. That last repair is what PPO is.

∇J = E[∇log π(a|s)·A(s,a)], the score-function estimator. Any baseline b(s) subtracted from the return leaves the gradient unbiased while changing its variance, which is why the advantage A = Q − V is used rather than Q: it is the same gradient with much less noise in it.