Planning / Methods
verifiedHierarchical RL
Deciding every muscle twitch and deciding where to go are different problems at different timescales. Hierarchical methods split them: a high-level policy picks a goal every so often, a low-level policy pursues it step by step. Credit then travels back over a handful of decisions rather than thousands, which is what makes long-horizon tasks learnable at all.
The persistent difficulty is that the two levels are learning against each other — the low level's behaviour is the high level's transition function, so it is non-stationary from above. Fixing the low level early stabilises learning and caps the ceiling; letting both move risks neither converging. How goals are represented matters more than the hierarchy itself, and learned goal spaces remain the unsolved part.
An option is a triple: a policy, an initiation set, and a termination condition. Over options the problem is a semi-MDP whose transitions span variable durations, so the discount over an option lasting k steps is γ super k rather than γ. That single change is what shortens the credit assignment path — a hundred primitive steps become a handful of option-level ones.
option-length holds 50% of the budget; rest holds the remaining 50%.
Steps a single option runs for against the decision that launched it, in steps. Drag the option length up to watch the high level decide less often — which shortens credit assignment and hands more of the behaviour to a policy it cannot see inside.
Reviewed by opendroid · 2026-08-18
- arXiv:1703.01161 — FeUdal Networks for Hierarchical Reinforcement Learning