Foundations / Training
verifiedExposure Bias
The consequence of Teacher Forcing: a model has only ever seen correct prefixes, so when its own output drifts it is somewhere it has no experience of, and it has no idea how to recover. One wrong token early can send a whole generation somewhere strange, and the model will continue confidently.
The proposed fixes have a poor record relative to their prominence. Gradually replacing ground-truth tokens with the model's own during training addresses the symptom and introduces its own inconsistency in what the model is being fit to. Sequence-level training with a reward optimises the thing you care about and is much harder to make stable. What actually reduced it in practice was scale and better data, which is unsatisfying as an explanation and is what happened.
Errors compound because the conditioning is recursive: a token sampled at step t is conditioned on for every step after, so a deviation does not average out but propagates. The reachable set of prefixes grows with length while the training distribution covers a fixed sliver of it, so the share of generation time spent off-distribution rises with the length of the output — which is why long generations degrade in a way short ones do not.
off-distribution-steps holds 17% of the budget; rest holds the remaining 83%.
Generation steps spent on a prefix unlike anything in training, against steps still close to it, in steps. Drag the off-distribution count up to watch it take the generation — it rises with output length, which is why long outputs degrade and short ones do not.
Reviewed by opendroid · 2026-08-18
- arXiv:1511.06732 — Sequence Level Training with Recurrent Neural Networks
- arXiv:1506.03099 — Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks