Compare

Bi-encoder vs Cross-encoder

Two ways embeddings power retrieval and reranking

The verdict

Bi-encoders precompute embeddings and search fast; cross-encoders score query and document together for higher quality. Production usually runs bi-encoders for retrieval and cross-encoders as a reranker on top.

Bi-encoder

Glossary →

Encodes the query and each document independently into fixed-size vectors, then compares by cosine or dot-product. Fast to search over millions of docs.

Best for: First-stage retrieval at scale.

Cross-encoder

Glossary →

Passes query and document together through a transformer and emits a single relevance score. Much more accurate, but you can't precompute doc embeddings.

Best for: Reranking a small candidate set.

At a glance

Bi-encoder vs Cross-encoder: dimension-by-dimension comparison
Dimension	Bi-encoder	Cross-encoder
Precompute	Yes (doc embeddings)	No
Query-doc interaction	None (dot product)	Full attention
Scale	Millions of docs	Dozens per query
Latency per query	Milliseconds	Tens to hundreds of ms
Quality	Baseline	Higher (used to rerank)
Best for	First-stage retrieval	Reranking short lists

Key differences

1Bi-encoders precompute embeddings; cross-encoders re-score every pair
2Bi-encoders scale to millions of docs; cross-encoders scale to dozens per query
3Cross-encoders capture interaction between query and doc; bi-encoders only see them separately
4Retrieval quality gap: cross-encoder rerank usually adds 10-20 points of MRR/nDCG
5Latency: bi-encoder is O(log N); cross-encoder is O(k) per query where k is the shortlist size

In the interview

What they're really testing

Whether you know why production RAG uses both, and which one goes first.

Say this

A bi-encoder encodes the query and every document independently and compares by dot product, so you can search millions of docs in milliseconds. A cross-encoder feeds the query and doc together through a transformer, capturing their interaction and scoring much more accurately, but you can't precompute anything. Production RAG uses the bi-encoder to fetch a shortlist and a cross-encoder to rerank it; that combination usually adds ten to twenty points of ranking quality for a fixed latency budget.

Traps to sidestep

Using a cross-encoder for first-stage retrieval
Ignoring reranking and blaming embeddings for poor RAG quality
Confusing bi-encoder with 'bidirectional encoder' (unrelated)

How to choose

If searching over a large corpusBi-encoder

If reranking a shortlist of ~10-100Cross-encoder

If you need per-pair scoring qualityCross-encoder

If latency budget is under 10 msBi-encoder

Retrieve with a bi-encoder, rerank with a cross-encoder. That's the production RAG default.

Common misconceptions

Myth: Better embeddings remove the need for reranking.

Reality: Cross-encoders capture interactions no static embedding can. A rerank stage still helps even the best embedding models.

Myth: Cross-encoders can search at scale.

Reality: They must score every candidate freshly, so a full-corpus scan is O(N) per query. That's why they only rerank a shortlist.

Memory aid

Bi-encoder: quick sort of the pile. Cross-encoder: carefully compare each finalist to the request.

Can you combine them?

This is the standard combo. Retrieve top-100 with a bi-encoder, rerank to top-5 with a cross-encoder, feed those to the LLM.

Related comparisons

Fixed-sizevsSemanticvsRecursive

Chunking strategies: fixed vs semantic vs recursive

Three ways to slice documents before they hit the vector store

Hallucination MitigationvsContext Window Management

Hallucination Mitigation vs Context Window Management

Two critical challenges in production LLM systems

Long contextvsRAG

Long context vs RAG

Show the model everything, or choose what it should see

RAGvsFine-tuning

RAG vs Fine-tuning

Two approaches to giving LLMs domain specific knowledge

RetrievervsReranker

Reranker vs Retriever

The two stages of production RAG, and why you need both

Semantic (Embedding) SearchvsKeyword (BM25) Search

Semantic Search vs Keyword Search

Dense vector similarity vs sparse term matching for retrieval