When would you reach for a cross-encoder over a bi-encoder?

A **cross-encoder** is a single transformer that takes a query and a candidate document concatenated as one input, and outputs one relevance score. There are no separate query and document vectors at any point. Contrast with a bi-encoder, which encodes the query and the document independently into vectors and compares them with cosine similarity. Bi-encoders let you precompute and index all the document vectors once, then do fast vector search at query time. Cross-encoders cannot precompute anything. Every new query means a fresh transformer forward pass per candidate. Over a million documents that is a million forward passes per query, which is far too slow for first-stage retrieval. In practice you use both in a two-stage pipeline. The bi-encoder retrieves the top 50 or top 100 candidates from the full corpus. Then the cross-encoder reranks just those candidates and you keep the top 5 or top 10. You get bi-encoder speed for the wide scan and cross-encoder quality for the final ranking.

**Cross-encoders** trade indexability for quality. The query and document are concatenated into one input and pushed through a transformer that does full bidirectional attention between every query token and every document token. The output is a single relevance score. $$s(q, d) = \text{CrossEncoder}([q\,;\,d])$$ That tight token-level interaction is what makes cross-encoders higher quality per pair than bi-encoders, but it is also why they cannot scale. Scoring a query against N documents costs N full transformer passes. At a million documents that is unworkable. The canonical use is the second stage of a two-stage retrieval pipeline. The bi-encoder retrieves the top 50 to 200 candidates via ANN search over precomputed vectors, then the cross-encoder rescores just those candidates and returns the top K (typically 5 to 10). Latency cost of the rerank stage is roughly 30 to 100ms for 100 pairs on a modern reranker like Cohere rerank-3.5, Voyage rerank-2.5, or BGE-reranker-v2-m3. Quality gains over the bi-encoder alone are typically 5 to 20 percent NDCG@10 on hard benchmarks like BEIR. Use a cross-encoder when recall at small K is materially below recall at large K. Skip it when your latency budget is sub-30ms or when recall is already saturating.