Dense Retrieval

Keyword search fails when the question and the answer use different words. Dense retrieval encodes both into vectors and compares meanings rather than strings, so a question about how to stop a car can find a passage about brakes. The price is that closeness is something the encoder had to learn, and it only learned it for the languages and domains it was trained on.

Two encoders, often sharing weights, map queries and passages into one space; retrieval is nearest-neighbour search over passage vectors computed ahead of time. Because a passage is encoded before any query exists, the encoder cannot model interaction between the two — a structural ceiling, and the one reranking exists to lift. Training uses the other passages in the batch as negatives, so batch size is part of the objective rather than a tuning knob.

score(q,p) = E sub _q (q)·E sub _p (p), a single dot product, so N passages are scored with one matrix multiply against a precomputed matrix. Training is softmax cross-entropy over one positive and the batch's other passages — the same InfoNCE objective contrastive learning uses, which is why the score matrix comes out diagonal-dominant.