Vision-Language-Action Model

Take a model that already understands images and instructions, and give it a third output: what to do with a body. The bet is that most of what a robot needs to know about the world was learnable from the internet, and only the acting part has to come from robots — which are slow, expensive and scarce.

Actions are usually emitted as tokens from the same vocabulary head, which is what lets web pretraining transfer at all rather than being discarded at the last layer. The evidence for transfer is semantic generalisation — instructions naming objects the robot never manipulated — rather than better control on familiar tasks. Inference latency is the binding constraint, because a large model at 3 Hz is not a controller.

Discretise each action dimension into bins and predict them as tokens, so the policy is a language model conditioned on images and an instruction. Robot data is the scarce factor by several orders of magnitude, which is why the scaling that matters is not model size but how much of the model can be pretrained on data that is not robot data.