Why do cross-encoder rerankers outperform dual-encoder retrievers at picking the most relevant chunks, given both models can be trained on the same data?

Attempt it

Quick reference

TL;DR

A cross-encoder reads the query and document together in one forward pass, so self-attention can cross between them. A dual-encoder embeds each in isolation and only compares afterward.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine two ways to judge whether a job applicant fits a role. In the first way, two people separately write a one paragraph summary, one of the job, one of the applicant, then a clerk who never read either resume just checks how similar the two paragraphs sound. That's a dual-encoder: fast, but it squeezes everything into a summary before comparing. In the second way, a hiring manager reads the job description and the resume side by side, line by line, noticing that this specific skill matches that specific requirement. That's a cross-encoder: slower, because someone has to read every job applicant pair fresh, but far more accurate because the comparison happens at the level of actual words, not pre-written summaries. RAG uses the fast clerk to shortlist, then the careful manager to rank that shortlist.

Key concepts

RAG system design Embeddings Self-attention Vector databases

Quick scan · 2–4 min

Everything important, quickly.

3 min: contrast joint vs independent encoding, explain why pooling loses token interactions, derive the per-pair cost, then justify the two-stage retrieve then rerank funnel and name a production reranker.

Code example

python

from sentence_transformers import SentenceTransformer, CrossEncoder

# Stage 1: dual-encoder retrieval (embeds query and docs independently)
bi = SentenceTransformer("BAAI/bge-base-en-v1.5")
q_vec = bi.encode(query, normalize_embeddings=True)
candidates = ann_index.search(q_vec, top_k=100)  # cosine over precomputed doc vectors

# Stage 2: cross-encoder rerank (scores each (query, doc) pair jointly)
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, c.text) for c in candidates]
scores = ce.predict(pairs)          # one joint forward pass per pair
reranked = [c for _, c in sorted(zip(scores, candidates), reverse=True)][:5]

Side-by-side

Property	Dual-encoder (bi-encoder)	Cross-encoder
Encoding	Query and doc embedded separately	Query and doc concatenated, encoded jointly
Interaction	One dot product of pooled vectors	Token-level self-attention across the pair
Scoring	Cosine / dot-product similarity	Relevance score from a regression head
Doc vectors precomputable	Yes, index once and reuse	No, fresh forward pass per query-doc pair
Cost at query time	1 embed + ANN lookup	O(candidates) forward passes
Role in RAG	First-stage retrieval over millions	Second-stage rerank over top-k

Real-world examples

Real products, models, and research that use this idea.

Cohere Rerank 3.5 is a hosted cross-encoder reranker that scores query-document pairs jointly over a candidate set from any first stage retriever.
BGE-reranker-v2-m3 from BAAI is an open weight cross-encoder widely dropped into LlamaIndex and LangChain RAG stacks for the rerank step.
Voyage rerank-2 and Jina Reranker v2 are production cross-encoders that take a query plus top-k candidates and return relevance ordered chunks.
Sentence-Transformers ships cross-encoder checkpoints (ms-marco-MiniLM) used as the canonical reranking baseline on the MS MARCO passage task.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you choose the candidate set size (top-k) that the cross-encoder reranks?

Sign in or go Pro to see the approach for this follow-up.

QWhere do late-interaction models like ColBERT sit between the two extremes?

Sign in or go Pro to see the approach for this follow-up.

QYour reranker barely improves results. How do you diagnose whether the problem is the reranker or the retriever?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.