Model Merging

Take two models fine-tuned from the same base for different things, average their weights, and get one model that does both. It should not work — nothing constrains the average of two good solutions to be a good solution — and it does, often well enough to skip retraining entirely.

The condition that makes it work is a shared ancestor. Fine-tunes of one base stay in the same region of the landscape, so the straight line between them does not leave the low-loss basin. Two models trained from different random initialisations have no such relationship, and averaging them produces something at chance — the difference between the two cases is the whole subject, and confusing them is the most common way people conclude merging does not work.

Fine-tuning moves weights a small distance relative to the base, so two fine-tunes differ from each other by roughly the sum of two small displacements. As long as the loss surface is close to flat over that region, the loss at the midpoint is close to the average of the losses — which is a statement about curvature and is why merging degrades as the fine-tunes get further apart. Nothing here promises anything about models that were never near each other.