Experience Replay

Keep a small sample of old data and mix it into every batch. It is the least clever method in continual learning and the one that usually wins, which is worth sitting with rather than explaining away. A few percent of the original data recovers most of what would otherwise be lost.

It also assumes you are allowed to store the data, which privacy and licensing frequently forbid — and that constraint, not accuracy, is why the other families exist. Generative replay substitutes samples from a model of the old data and inherits that model's failures, which compound over a long sequence in the same way synthetic data does.

Minimise the new task's loss plus the loss on a stored subset, which is a sampled approximation to the joint objective the problem actually wants. Because it approximates the true objective directly rather than through a proxy on parameters, its error falls with buffer size in a way regularisation penalties have no equivalent of — which is the cleanest explanation for why it wins.