Merge Interference

Merging two models degrades both a little. Merging eight degrades all of them a lot. The reason is that the fine-tunes do not agree about what to do with the same weights — one wants a coordinate raised, another wants it lowered — and averaging splits the difference in a way that satisfies neither.

The mitigations all attack the same thing from different angles. Trim each task vector to its largest-magnitude coordinates, since most of a fine-tune's displacement is noise. Resolve sign conflicts by majority before averaging, so opposing directions do not cancel to nothing. Drop a large random fraction of the vector and rescale the rest, which works surprisingly well and is evidence for how redundant these displacements are.

Interference grows with the number of models merged because the chance that any given coordinate is contested rises with each one added. If each task vector touches a fraction of coordinates, the expected number of contested coordinates grows roughly with the square of the model count while the useful signal grows only linearly — so there is a count past which adding a model makes the merge worse, and it arrives sooner than the linear intuition suggests.