Reinforcement / Foundations
verifiedCredit Assignment
Something good happened. Which of the things you did made it happen? The question sounds simple and is the central difficulty of learning from delayed feedback — a reward arriving at move two hundred says nothing about which of the two hundred mattered, and most of them did not.
It appears in two forms that get the same name. The structural one asks which parameter to change, and Backpropagation answers it exactly by the chain rule. The temporal one asks which earlier action deserves the reward, and nothing answers it exactly — value functions, eligibility traces and advantage estimates are all approximations, and Process Supervision exists because the cheapest solution is to stop needing it and label each step.
The difficulty scales with the delay. With one reward at the end of a trajectory of length T, the feedback carries one number for T decisions, so the information per decision falls as 1/T while the number of plausible explanations for the reward grows with it. That is the arithmetic behind sparse-reward tasks being hard in a way that is not about the reward being rare — it is about the reward being uninformative once it arrives.
trajectory-length holds 29% of the budget; rest holds the remaining 71%.
Decisions too far back for one end-of-episode reward to speak to, against the recent ones it plausibly does, in decisions. Drag the trajectory length up to watch the unattributable share take over — the reward is not rarer, it is less informative about any one thing you did.
Reviewed by opendroid · 2026-08-18
- arXiv:1806.07857 — RUDDER: Return Decomposition for Delayed Rewards
- arXiv:1506.02438 — High-Dimensional Continuous Control Using Generalized Advantage Estimation