Task Interference

Training on several things at once is not always better than training on each. Tasks can help each other, and they can pull the shared parameters in opposing directions — the same mechanism that makes multi-task learning attractive makes it fail, and which one happens depends on the tasks rather than on the setup.

The diagnostic is gradient conflict: when per-task gradients point in opposing directions, the update serves neither. Methods that project one gradient off another help modestly and cost a backward pass per task. The larger lever is usually capacity — interference falls when tasks are given somewhere separate to live, which is what adapters and mixtures of experts do without calling it that.

Two tasks conflict at a point when the cosine between their gradients is negative, so the summed step increases at least one loss. Conflict is not a property of a task pair but of a point in parameter space, which is why it appears and disappears during training and why measuring it once says little.