Reinforcement / Methods
verifiedQ-Learning
Learn a number for every state-action pair saying how good that action is there, and always update as though you will act optimally afterwards — even when you did not. That gap is the point: it lets an agent learn the best policy while behaving badly enough to explore, which is why it was the first method to feel like it could learn anything.
Off-policy and tabular in its original form, which stops working the moment the state space is large. Deep Q-networks replaced the table with a network and needed two repairs to stay stable — a replay buffer to break the correlation between consecutive samples, and a frozen target network so the thing being chased does not move every step.
Q(s,a) ← Q(s,a) + α[r + γ·max sub a′ Q(s′,a′) − Q(s,a)]. The max is what makes it off-policy and also what makes it overestimate: taking a maximum over noisy estimates is biased upward, which is the whole reason double Q-learning exists.
replayed-updates holds 75% of the budget; rest holds the remaining 25%.
Updates drawn from the replay buffer against those taken from the episode just played, in updates. Drag the replay ratio to watch consecutive samples stop dominating what is learned from.
Reviewed by opendroid · 2026-08-17
- arXiv:1312.5602 — Playing Atari with Deep Reinforcement Learning
- arXiv:cs/9605103 — Reinforcement Learning: A Survey
Origin · not linkable
- Watkins & Dayan 1992 — Q-learning · Machine Learning 8(3) · doi:10.1007/BF00992698