Value Function

How good is it to be here? A value function answers that with one number per state — the total reward you expect to collect from it under your current behaviour. Once you have it, choosing an action stops needing foresight: take whichever one leads somewhere worth more.

V(s) values a state, Q(s,a) values a state-action pair, and the second is what you can act on directly without a model of the world. Both are defined relative to a policy, so improving the policy changes the values, which changes the best policy — the circularity that every method here is a way of resolving.

V super ^π (s) = E[Σ sub _t  γ super ^t  r sub _t  | s sub ₀  = s], and the Bellman equation makes it recursive: V super ^π (s) = E[r + γV super ^π (s′)]. That one step of self-reference is what allows learning from a single transition instead of from a whole episode, and it is the reason a table of values can be improved before anything has finished.