Vision-Language Model

A language model that can see is mostly a language model. A vision encoder turns the image into vectors, a small bridge maps them into the language model's embedding space, and from there the image is just more tokens in the context. The heavy lifting stays where it already was.

Freeze a pretrained vision encoder and a pretrained language model, train the connector between them, and optionally add cross-attention layers so text can attend to image features. Training the bridge alone is far cheaper than training either side, which is why capability arrives faster here than the compute would suggest.

Project visual features V ∈ ℝ super ^{N×d sub _v}   into ℝ super ^{N×d sub _model}   and either prepend them to the token sequence or attend to them from interleaved cross-attention layers. Prepending costs Θ((N+n)²) attention over the joined sequence; cross-attention costs Θ(n·N) and keeps the two lengths independent.