Any-to-Any Model

One model that takes any mixture of text, images and audio and produces any mixture back, rather than a text model with an image encoder bolted on. The appeal is that a single sequence of tokens makes no distinction between reading an image and writing one — the same machinery does both, and interleaving them costs nothing extra.

It requires every modality to become discrete tokens, so images pass through a vector quantiser and audio through a codec, and the model is then an ordinary autoregressive transformer over a mixed vocabulary. The difficulty is training stability: mixed-modal sequences have very different token statistics per modality, and the standard recipe needs normalisation and initialisation changes that a text-only model does not.

The tokenisation is where the cost sits. An image at useful resolution is hundreds to thousands of tokens against a sentence's tens, so a mixed sequence is dominated by whichever modality tokenises most expensively — and it is not the one carrying most of the meaning. That imbalance drives the sequence length, and through the quadratic in attention it drives the compute, which is why every practical system compresses the image side before the model sees it.