Catastrophic Forgetting

Train a network on a second task and it does not merely get worse at the first — it can lose it almost entirely, within a few hundred steps. Nothing about the architecture protects earlier learning, because the weights that encoded it are the same weights the new gradient is free to move.

This is the risk under every fine-tune in this graph, and it is why instruction tuning degrades pretrained abilities and why RLHF is kept near a reference policy. It is worse when the new data is narrow: a small, homogeneous fine-tuning set moves a few directions a long way, which is exactly the shape that overwrites. Mixing a slice of the original data back in is the crude fix and it works.

Sequential training minimises the new task's loss with no term for the old one, so the solution drifts to wherever the new gradient leads — and the old loss is unconstrained along that path. Nothing is forgotten in a storage sense; the parameters simply move out of the region where the old task was solved, which is why a small penalty pulling back toward that region recovers so much.