Modality Fusion

Two streams of information have to become one prediction, and where they meet decides most of what the system can do. Join them at the input and the model can relate a pixel to a word from the first layer. Join them at the output and each stream is processed in ignorance of the other, then reconciled by a vote.

Late fusion is cheapest and is what a two-tower retrieval system does — each side encodes independently, so both can be precomputed. Early fusion tokenises every modality into one sequence and lets ordinary attention do the work, which is the most capable and the most expensive because nothing can be cached. Cross-attention sits between: one modality stays a sequence and the other is attended into it, which is how most vision-language models are built and why they can reuse a frozen image encoder.

The cost separates them. Late fusion computes each stream once and combines with something cheap, so cost is additive in the modalities. Early fusion concatenates into one sequence, and attention is quadratic in its length, so joining a thousand image tokens to a thousand text tokens costs four times what either did alone rather than twice. That factor is why early fusion arrived last and why it arrived with a compression step in front of it.