Explain when you'd add a cross-encoder on top of an embedding-based retriever and why

Short answer·Medium·4.0 · 0·~3 min·Asked atCohere Perplexity

Attempt it

Given a working bi-encoder retrieval system at recall@100 = 0.92, when does it make sense to add a cross-encoder reranker, and what is the latency/quality tradeoff?

Free · 2 AI evals / day

Bi-encoder retrievers are fast and broad: they find the right neighborhood in the document corpus. But the ranking inside that neighborhood is often imperfect. The right document might be at position 47 in the top 100, when you needed it at position 1. **Cross-encoder rerankers** fix that ranking problem. You take the top 50 or top 100 candidates the bi-encoder returned, score each one with the cross-encoder, and reorder by that score. The cross-encoder reads the query and each candidate document together with full transformer attention, which gives it sharper relevance judgment than cosine similarity on precomputed vectors. The decision rule is simple. Look at recall@100 and recall@10 on your eval set. If recall@100 = 0.92 but recall@10 = 0.68, you have a ranking problem the reranker can fix. If recall@10 is already 0.90, skip the reranker. The cost is roughly 30 to 100ms of extra latency per query for a 100-pair rerank in 2026, using models like Cohere rerank-3.5 or BGE-reranker-v2-m3. The quality gain is 5 to 20 percent NDCG@10 on hard benchmarks.

**The trigger is the recall gap.** A bi-encoder retriever that posts recall@100 = 0.92 has found the right document in the top 100 for 92 percent of queries. If recall@10 for the same retriever is only 0.68, then for 24 percent of queries the right document is in the bi-encoder's top 100 but not in its top 10. That is exactly the gap a cross-encoder reranker closes. **The mechanism.** The cross-encoder takes the bi-encoder's top 100 candidates, scores each against the query with a full transformer pass over the concatenated input. Returns the top K by that new score. Typical rerank candidate set is 50 to 200; typical final K is 5 to 10. **The numbers.** In 2026, a 100-pair rerank with a modern model (Cohere rerank-3.5, Voyage rerank-2.5, Jina reranker v2, BGE-reranker-v2-m3) takes 30 to 100ms on a single GPU with batching. Quality lift is typically 5 to 20 percent NDCG@10 over the bi-encoder alone, depending on the bi-encoder strength and benchmark. **When to skip.** If recall@10 is already 0.90+, the reranker adds latency for no win. If your latency budget is sub-30ms, you cannot afford the rerank stage. If your LLM consumes a large K (top 50) and is robust to ordering within that K, fine-grained reranking matters less. When to revisit recall first. If recall@100 is below 0.85, the reranker has a low ceiling. Improving the bi-encoder (better model, hybrid with BM25, query expansion) often beats adding rerank in that regime.

Quick reference

TL;DR

Add a cross-encoder when recall@100 is healthy but recall@10 is not; it rescores the top 50 to 100 candidates for a 5 to 20 percent NDCG lift at 30 to 100ms extra latency.

Memory aid

Easy to grasp

Picture a job recruiter with 100 promising resumes from a keyword filter. The filter is fast but not smart. To pick the best 5, the recruiter actually reads each resume carefully, that is slow but accurate. You only do the careful read on the 100 that passed the filter, not the 100,000 resumes in the database. The cross-encoder is the careful read. The bi-encoder is the keyword filter. You add the careful read when the filter is keeping the right people in the pile but ordering them badly.

Key concepts

Embeddings

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 to 8 min: recall@K diagnosis + two-stage pattern + candidate set sizing + 2026 reranker models + latency math + when not to use.

Real-world examples

Real products, models, and research that use this idea.

Cohere rerank-3.5 (2025) is the default managed reranker in 2026 production RAG stacks.
Voyage rerank-2.5 pairs with voyage-3-large embeddings in Anthropic-aligned stacks.
BGE-reranker-v2-m3 from BAAI is the most-deployed open-weight reranker, self-hosted on L4 or A10G GPUs.
Pinecone's managed inference offers built-in rerank as a hop after vector search, exactly this two-stage pattern.
Anthropic's contextual retrieval guide explicitly recommends two-stage retrieve plus rerank as the default for production RAG with Claude.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you build the eval set you need to measure recall@K honestly?

QWhen would you use late interaction (ColBERT) instead of a full cross-encoder reranker?

QWhat is the largest rerank candidate set that is ever worth it?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Adding a reranker without first measuring the gap between recall@100 and recall@10. If recall@10 is already high, the reranker adds latency for no win.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.