Cross-Modal Retrieval

Search images with a sentence, or find the caption that matches a picture. It needs the two modalities to live in one space where distance means the same thing across them — which is a stronger requirement than either modality having a good space of its own, and it is what the training has to produce.

CLIP is one way to get that space and not the only one; the retrieval question is what you do with it. The serving shape matters more than the objective: encoding each side independently lets the image side be precomputed and indexed, which is what makes search over millions of images possible, while a model that attends across both at scoring time is more accurate and can only rerank. Almost every deployed system is the first stage feeding the second.

A shared space is only as good as the alignment between the two distributions in it, and there is a specific failure worth naming: the modality gap. Trained image and text embeddings occupy separate cones rather than interleaving, so a text query is closer to every other text than to any image, and retrieval works only because the ranking within the image set is still correct. The absolute similarities are therefore not comparable across modalities, and thresholding on them rather than ranking is the mistake this invites.