Mode Connectivity

Two networks trained from different random seeds end up in what look like separate valleys — but you can often find a curved path between them along which the loss never rises. The valleys are connected. What looked like isolated solutions is one region with a complicated shape.

The practical uses are direct: averaging weights along such a path gives an ensemble at the cost of one model, and the same fact explains why weight averaging works at all rather than producing nonsense. The connecting path is curved — the straight line between two solutions does have a barrier — which is why simple interpolation between checkpoints fails while a fitted path does not.

Fit a curve φ(t) from θ sub ₁  to θ sub ₂  minimising the expected loss along it; the finding is that low-loss curves exist between independently trained solutions. Permutation symmetry explains much of it: two networks differing by a hidden-unit permutation are the same function, and accounting for that permutation removes most of the barrier on the straight path.