Advantage Estimation

Was this action better than what I would normally have done here? That is the advantage — the value of an action minus the value of the state it was taken in — and it is a far more useful learning signal than the raw return, because it strips out how good the situation was and leaves only what the choice contributed.

Estimating it is a bias-variance dial with a name. Use the full return and the estimate is unbiased and extremely noisy, because it carries every random thing that happened afterwards. Use one step and a learned value function and it is low-variance and biased by however wrong that function is. Generalised advantage estimation interpolates between them with one parameter, and that parameter is the actual knob practitioners turn.

The interpolation weights an exponentially decaying sum of k-step estimates by λ. At λ = 0 it is the one-step temporal-difference error — biased, quiet. At λ = 1 it is the Monte Carlo return — unbiased, loud. Every value between trades the two smoothly, and the reason a middle setting wins is that neither endpoint is where the total error is smallest: bias and variance are both costs, and the sum of them has an interior minimum.