Modality Imbalance

Train a model on two modalities and one of them usually wins. If text alone predicts the label well enough, the model learns to read and stops looking at the image — not because vision is useless but because it was slower to help, and gradient descent takes the shortcut it finds first.

It shows up as a multimodal model that barely degrades when a modality is removed at test time, which is the diagnostic worth running and rarely is. The mitigations all slow the dominant modality down or speed the other up: separate learning rates per encoder, dropping a modality at random during training, or pretraining the weaker encoder alone before joining. Each is an admission that joint training does not balance itself.

The mechanism is convergence rate rather than capacity. Whichever modality reduces the loss fastest early gets the gradient, and once the shared layers are organised around it the other modality's features arrive into a representation already shaped against them — so the advantage compounds instead of evening out. That is why the imbalance grows with training rather than washing out, and why the interventions have to act early to act at all.