Why has the 2026 default become wide retrieval plus a cross-encoder rerank?
Stage 1 is a fast bi-encoder optimized for recall over a wide candidate pool; stage 2 is a slow cross-encoder that scores query and chunk jointly to sharpen precision on the survivors.
Picture hiring for one open seat at a software company. You do not interview every applicant in depth. First, a resume scan sweeps thousands of resumes and pulls maybe fifty that look plausible. That sweep is fast and forgiving on purpose: you want the right candidate in the pile, even if they are not ranked first. Then a smaller panel does deep interviews with those fifty and picks the actual top three. Two passes with two different filters do a better job than one pass with one filter that has to be both fast and thorough. RAG retrieval works the same way. Vector search is the resume scan. The reranker is the interview panel.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
If you only retrieve once, you have to pick between two bad options. A fast bi-encoder index can search the whole corpus quickly but its scores are coarse, so its top-3 is often wrong in the order that matters. A cross-encoder is precise but cannot be run over a million chunks per query; the inference cost is prohibitive.
The two-stage architecture resolves the conflict by giving each model the job it is good at. Stage 1 casts a wide net cheaply. Stage 2 sorts the catch carefully. This card walks through why the split is the 2026 default, how the two stages differ structurally, and how to tune the parameters that fall out of the split.
Bi-encoders versus cross-encoders
The two architectures encode queries and chunks very differently, and that difference is the whole point.
A bi-encoder runs query and chunk through the model independently. Each gets its own embedding vector. Similarity is measured downstream with cosine or dot product. Because the chunk encodings are independent of the query, you can precompute them once for the entire corpus and store them in an ANN index. At query time, you only embed the query and do an approximate nearest-neighbor lookup. This is fast (milliseconds) and reusable across queries.
A cross-encoder runs the query and the chunk through the model jointly, concatenated into a single input. Attention can compare every query token against every chunk token. The output is a single scalar score that reflects how well the pair matches. This is much more accurate per pair but cannot be precomputed: every (query, chunk) pair requires a fresh forward pass.
The structural consequence
Bi-encoders are corpus-side cacheable; cross-encoders are not. That asymmetry is what forces the two-stage shape. You cannot afford a cross-encoder over a million chunks per query; you can afford one over fifty.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from anthropic import Anthropic
import cohere
vector_db = ... # Faiss / Pinecone / Qdrant client
co = cohere.Client()
def retrieve(query: str, k1: int = 30, k2: int = 5) -> list[str]:
# Stage 1: wide bi-encoder recall
candidates = vector_db.search(query, top_k=k1)
# Stage 2: cross-encoder rerank to top k2
docs = [c.text for c in candidates]
rerank = co.rerank(model="rerank-3.5", query=query, documents=docs, top_n=k2)
return [docs[r.index] for r in rerank.results]Real products, models, and research that use this idea.
- Cohere Rerank 3.5 is the default reranker for many production RAG stacks in 2026; pairs naturally with Cohere Embed v4 or any dense index.
- BGE Reranker v2 (from BAAI) is the open-weight 2026 baseline; runs well alongside BGE-M3 embeddings.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you tune k1 and k2 for a high-recall medical-literature search versus a low-latency support bot?
Medical wants k1 large (50-100) and a strong cross-encoder; support bot wants k1 small (15-20) and rerank to top 3 with a faster model.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping the rerank because vector search already returns ranked results. Bi-encoder scores correlate weakly with true relevance past the top few; reranking is what makes top-5 actually be top-5.
60 second bullets to scan on the way to the call.
Define bi-encoder versus cross-encoder in one sentence each
State why stage 1 prioritizes recall and stage 2 prioritizes precision
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.