Model-Based RL

Model-free methods learn what to do; model-based methods learn how the world works and then work out what to do. The second needs far less experience, because a single transition teaches the model something usable everywhere rather than nudging one value estimate. What it buys in sample efficiency it gives back in a new way to be wrong.

The failure is model bias, and it compounds with rollout length rather than staying constant. That single fact explains most of the design space: branch short rollouts from states you actually visited, weight by ensemble disagreement, or use the model only for value expansion rather than for the policy itself. When a paper reports model-based methods beating model-free at ten times fewer samples, the rollout length is usually where to look first.

With a model error ε per step, the return estimated from an h-step rollout carries error growing roughly with h, so total error trades against the variance reduction longer rollouts buy. The optimum is short and non-zero — a one-step model is nearly useless and a hundred-step one is fiction — which is why h is the hyperparameter these methods live and die on.