Reinforcement / Methods
verifiedTemporal Difference Learning
Wait until the end of the game to learn what your moves were worth and you learn slowly and rarely. Temporal difference learning updates on every step instead, by comparing what you predicted with what you predicted one step later — bootstrapping off your own estimate rather than waiting for the truth.
The update is driven by the TD error: reward plus discounted next-state value, minus current value. It learns online and from incomplete episodes, and it is biased in exchange for far less variance than waiting for the full return. That bias-variance dial is the whole design space between one-step TD and Monte Carlo.
δ sub t = r sub t + γV(s sub t+1 ) − V(s sub t ), applied as V(s sub t ) ← V(s sub t ) + α·δ sub t . TD(λ) interpolates: λ = 0 is one-step bootstrapping and λ = 1 is the full return, so the parameter is a direct handle on how much of the estimate is your own guess.
bootstrapped-steps holds 5% of the budget; rest holds the remaining 95%.
Steps whose value comes from the agent's own estimate against those backed by observed reward. Drag the bootstrapping up to trade the variance of waiting for the bias of guessing.
Reviewed by opendroid · 2026-08-17
- arXiv:cs/9605103 — Reinforcement Learning: A Survey
Origin · not linkable
- Sutton 1988 — Learning to Predict by the Methods of Temporal Differences · Machine Learning 3(1) · doi:10.1007/BF00115009