Reranking

The fast retriever has to score millions of passages, so it can only afford to look at the query and the passage separately. A reranker looks at a few dozen and reads them together with the question, which catches the cases where the words line up and the meaning does not. Expensive per passage, cheap overall, because so few are left.

A cross-encoder concatenates query and passage and runs a transformer over the pair, so every query token can attend to every passage token — the interaction a bi-encoder structurally cannot have. Nothing can be precomputed, so cost is linear in the number of candidates and it only makes sense on a shortlist. The gain over the retriever's own ordering is usually large, which is why nearly every serious stack has one.

A bi-encoder scores N passages with N dot products against precomputed vectors; a cross-encoder scores k with k forward passes over the joined pair, at Θ(k·(n sub _q  + n sub _p )²d). They are used together because the first is effectively constant per passage at query time and the second is not — the shortlist is the whole reason the second is affordable.