Actor-Critic

Two learners working together: an actor that chooses what to do and a critic that judges how good the situation is. The actor improves using the critic's opinion rather than waiting for the final score, and the critic improves from what actually happens. It is the marriage of the two families — policy gradients and value functions — and most modern methods are in it.

The critic supplies the baseline that policy gradients need, turning a whole-trajectory return into a per-step advantage. The cost is a second network to train and a second thing to go wrong: a critic that is wrong early sends the actor somewhere bad, and the actor's new behaviour then supplies the critic with data from that bad place.

The actor follows ∇log π(a|s)·A(s,a) with A estimated by the critic, commonly as a generalised advantage — an exponentially weighted average over n-step errors, with λ trading the critic's bias against the return's variance exactly as TD(λ) does. PPO is this with a clipped ratio in place of the raw gradient step.