Synthetic Data

Generate training data with a model instead of collecting it. It works well enough that small models trained on carefully generated textbook-style data outperform much larger ones trained on raw web text — and it fails in a specific way when a model is trained on its own output repeatedly, losing the tails of the distribution.

The distinction that matters is whether the generator knows something the learner does not. Distilling a stronger model, or filtering generations by an executable check, adds information; sampling from the model you are about to train adds none and removes variance. Verification is what makes it work, which is why code and mathematics — where correctness is checkable — are where synthetic data has gone furthest.

Repeated training on generated data narrows the distribution, because each round samples from an estimate and estimates lose the tails first. Successive generations converge toward a low-variance mode — model collapse — unless real data is retained in the mixture or generations are filtered by a signal outside the model.