Vision Transformer

Convolutions were assumed necessary for images because they build in the right assumptions: nearby pixels relate, and a cat is a cat wherever it appears. The vision transformer drops those assumptions and applies an ordinary transformer to image patches. It loses to convolutions on small datasets and beats them on large ones, because with enough data the assumptions can be learned rather than built in.

Patch embed, add positions, run a standard encoder stack, classify from a prepended token. Architecturally almost nothing is new — the contribution is the demonstration that nothing needed to be. That is what made one architecture serve text, images and audio, and it is the reason multimodal models are possible at all.

Identical to a transformer encoder over N = HW/P² tokens, so cost is Θ(N²d) in attention and Θ(Nd²) in projections. The inductive bias a convolution supplies for free — locality and translation equivariance — must instead be paid for in data, which is why the crossover with convolutional models happens at scale rather than everywhere.