Explain when you'd add a cross-encoder on top of an embedding based retriever and why

Short answer·Medium·4.0 · 0·~3 min·Asked atInfosys Replicate Turing·Relevant atElastic Hugging Face Neo4j Qdrant

Attempt it

Given a working bi-encoder retrieval system at recall@100 = 0.92, when does it make sense to add a cross-encoder reranker, and what is the latency/quality tradeoff?

Free · 2 AI evals / day

Bi-encoder retrievers are fast and broad: they find the right neighborhood in the document corpus. But the ranking inside that neighborhood is often imperfect. The right document might be at position 47 in the top 100, when you needed it at position 1. **Cross-encoder rerankers** fix that ranking problem. You take the top 50 or top 100 candidates the bi-encoder returned, score each one with the cross-encoder, and reorder by that score. The cross-encoder reads the query and each candidate document together with full transformer attention, which gives it sharper relevance judgment than cosine similarity on precomputed vectors. The decision rule is simple. Look at recall@100 and recall@10 on your eval set. If recall@100 = 0.92 but recall@10 = 0.68, you have a ranking problem the reranker can fix. If recall@10 is already 0.90, skip the reranker. The cost is roughly 30 to 100ms of extra latency per query for a 100-pair rerank in 2026, using models like Cohere rerank-3.5 or BGE-reranker-v2-m3. The quality gain is 5 to 20 percent NDCG@10 on hard benchmarks.

**The trigger is the recall gap.** A bi-encoder retriever that posts recall@100 = 0.92 has found the right document in the top 100 for 92 percent of queries. If recall@10 for the same retriever is only 0.68, then for 24 percent of queries the right document is in the bi-encoder's top 100 but not in its top 10. That is exactly the gap a cross-encoder reranker closes. **The mechanism.** The cross-encoder takes the bi-encoder's top 100 candidates, scores each against the query with a full transformer pass over the concatenated input. Returns the top K by that new score. Typical rerank candidate set is 50 to 200; typical final K is 5 to 10. **The numbers.** In 2026, a 100-pair rerank with a modern model (Cohere rerank-3.5, Voyage rerank-2.5, Jina reranker v2, BGE-reranker-v2-m3) takes 30 to 100ms on a single GPU with batching. Quality lift is typically 5 to 20 percent NDCG@10 over the bi-encoder alone, depending on the bi-encoder strength and benchmark. **When to skip.** If recall@10 is already 0.90+, the reranker adds latency for no win. If your latency budget is sub-30ms, you cannot afford the rerank stage. If your LLM consumes a large K (top 50) and is robust to ordering within that K, fine-grained reranking matters less. When to revisit recall first. If recall@100 is below 0.85, the reranker has a low ceiling. Improving the bi-encoder (better model, hybrid with BM25, query expansion) often beats adding rerank in that regime.

TL;DR

Add a cross-encoder when recall@100 is healthy but recall@10 is not; it rescores the top 50 to 100 candidates for a 5 to 20 percent NDCG lift at 30 to 100ms extra latency.

Memory aid

Easy to grasp

Picture a job recruiter with 100 promising resumes from a keyword filter. The filter is fast but not smart. To pick the best 5, the recruiter actually reads each resume carefully, that is slow but accurate. You only do the careful read on the 100 that passed the filter, not the 100,000 resumes in the database. The cross-encoder is the careful read. The bi-encoder is the keyword filter. You add the careful read when the filter is keeping the right people in the pile but ordering them badly.

Key concepts

Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Adding a cross-encoder is one of the most common reflex moves in RAG, and one of the most commonly wrong. The right framing is a recall diagnosis: figure out where retrieval quality is leaking, then decide if reranking is the right tool.

This deep dive covers how to read the recall@K curve, when the gap signals a reranker is the fix, how to size the rerank candidate set, the latency math in 2026, the 2026 reranker landscape, and the four common failure modes where adding a reranker does not help.

Mental model: a reranker closes the gap between recall@100 and recall@10. It does not raise recall@100. Diagnose the right gap first.

Reading the recall curve to diagnose the bottleneck

What recall@K measures

Recall@K is the fraction of queries for which the gold (correct) document is in the retrieved top K. A bi-encoder with recall@100 = 0.92 finds the right doc somewhere in the top 100 for 92 of every 100 queries.

The diagnostic shape

Measure recall at multiple K values and look at the curve:

Healthy retriever, bad ranker: recall@100 = 0.92, recall@10 = 0.68. The retriever is finding the right doc but ranking it low. A reranker fixes this.
Bad retriever: recall@100 = 0.72, recall@10 = 0.62. The retriever often does not find the right doc at all. A reranker has a low ceiling (0.72); fix the retriever first.
Saturated retriever: recall@100 = 0.96, recall@10 = 0.93. Little room for a reranker to add value. Often not worth the latency.

The trap

Teams that pattern-match on 'add a reranker' without this diagnosis end up in the saturated or bad-retriever regime, paying the latency cost for negligible gain. The rerank stage is only the right answer in the first shape.

Sizing the rerank candidate set

Latency math and the 2026 reranker landscape

Failure modes and when not to add a reranker

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Cohere rerank-3.5 (2025) is the default managed reranker in 2026 production RAG stacks.
Voyage rerank-2.5 pairs with voyage-3-large embeddings in Anthropic-aligned stacks.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you build the eval set you need to measure recall@K honestly?

Collect real queries from logs or user studies, label gold documents with human annotators or model-judges (cross-check with humans), and measure recall@K on a held-out split. Synthetic queries from an LLM are a starting point but tend to be easier than real traffic.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Adding a reranker without first measuring the gap between recall@100 and recall@10. If recall@10 is already high, the reranker adds latency for no win.

60 second bullets to scan on the way to the call.

Recall@K definition and how to compute it on an eval set
Why recall@100 vs recall@10 is the diagnostic gap for rerankers

Primary sources. Browse if you want the original framing.