Imitation Learning

Rather than specify a reward, show the behaviour you want and have the model copy it. It sidesteps reward design entirely, which is the hardest part of reinforcement learning in practice. The weakness appears the moment the learner drifts somewhere the expert never went — it has no demonstration for the situation its own mistake created.

Behaviour cloning is supervised learning on state-action pairs, and its errors compound quadratically in the episode length because each mistake moves the learner further off the expert's distribution. The fixes either collect corrective data where the learner actually goes, or match distributions rather than actions — which is what adversarial imitation does, treating the discriminator's output as a learned reward.

With a per-step error ε under the expert's state distribution, behaviour cloning's regret over an episode of length T grows as roughly ε·T² rather than ε·T, because errors move the state distribution as well as the action. Distribution-matching methods target that directly, minimising a divergence between occupancy measures rather than a loss on individual actions.