Teacher Forcing

During training, feed the model the correct previous tokens rather than the ones it just produced. It learns much faster this way — every position gets a clean prefix instead of compounding its own mistakes — and it never once practises the situation it will actually be in.

It is also what makes training parallelisable. Because the whole correct sequence is known in advance, every position can be predicted at once rather than one after another, which is the property that lets a transformer train on a thousand positions in a single forward pass. Sequential generation at inference is not an oversight; it is the price of a training procedure that was never sequential.

The model is fit to p(x sub _t  | x sub _<t ) with x sub _<t  drawn from the data, and used to sample from p(x sub _t  | x̂ sub _<t ) with x̂ sub _<t  drawn from itself. Those are different conditioning distributions, so the model is evaluated off the distribution it was fit on from the second token onward — and the gap widens with every step, because each sampled token moves the prefix further from anything the training set contained.