Multimodal / Objectives
verifiedCLIP
CLIP trains an image encoder and a text encoder together so that a picture and its caption land in the same place. Once they share a space, classification needs no classifier: encode the candidate labels as sentences and pick whichever sits closest. It learns from captions scraped at scale rather than from anyone labelling images by hand.
Two encoders, a contrastive objective over a batch of image-text pairs, and a shared embedding space. The result transfers to tasks it was never trained on, because the supervision was language rather than a fixed label set. It is also the component most multimodal systems build on — the image side becomes the eyes of a language model.
Maximise cosine similarity for matched pairs and minimise it for the B²−B mismatched ones, symmetrically over both directions. Zero-shot classification is argmax sub c cos(f sub img (x), f sub txt (prompt(c))) — the label set is a runtime argument rather than a trained output layer, which is what makes it open-vocabulary.
8 queries against 8 keys; a brighter cell means more of that query's attention went to that key. Each query sees itself and everything before it, and nothing after — the upper triangle is masked.
Image-text similarity across a batch, with matched pairs on the diagonal. Drag the batch size to watch the negatives multiply.
Reviewed by opendroid · 2026-08-04
- arXiv:2103.00020 — Learning Transferable Visual Models From Natural Language Supervision