A retrieve then rerank funnel pulls a wide pool of N candidates for recall, then a reranker trims them to the small top-k that actually go into the prompt.
Imagine hiring for one job. First you collect a big stack of resumes — maybe 100 — so you don't miss any good candidate. That wide net is the retrieval stage. Then a careful reviewer reads each one and picks the 5 best to interview. That careful reviewer is the reranker. RAG, short for Retrieval-Augmented Generation, works the same way: the first stage casts a wide net of N documents so nothing relevant slips through, and the reranker hands the model only the top-k that deserve a seat in the prompt.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Retrieval quality is the single biggest lever on RAG answer quality — most 'the LLM hallucinated' bugs are really 'the right chunk never made it into the prompt' bugs. The two-stage retrieve then rerank funnel is the standard production answer to that problem, and naming its two ends correctly is the entry ticket to talking about it.
The two numbers, N and k, encode a deliberate split of labor. The first stage is built for recall at corpus scale; the second is built for precision over a shortlist. This question looks trivial — fill in two letters — but the reason the funnel has two ends at all is the whole point. We'll cover what each stage does, why one cheap retriever can't do both jobs, and how the sizes of N and k are tuned in real systems.
Why retrieval is a funnel, not a single lookup
Picture searching a corpus of two million chunks for the three passages that answer a question. You face a tension. A model precise enough to rank those two million chunks perfectly — reading the query and each chunk together — would take far too long to run over the whole corpus on every query. A model fast enough to scan two million chunks in milliseconds is necessarily coarse.
The funnel resolves the tension by splitting the work. Stage one runs the fast, coarse retriever over everything and produces a wide candidate pool of N documents. Its only job is recall: get the relevant chunk into the pool, even if it lands at rank 40.
Stage two runs the slow, precise reranker over just those N candidates. Its job is precision: reorder the pool so the truly relevant chunks rise to the top, then keep the top-k. Wide then narrow. The fast stage handles scale; the slow stage handles quality. Neither could do the other's job alone, which is exactly why there are two ends to name.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Cohere Rerank is the canonical hosted second stage: retrieve N=100 with a vector DB, rerank, keep top-k=5
- LlamaIndex and LangChain both expose a similarity_top_k for the wide pool and a separate rerank node that trims to the final k
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you choose the value of N for the first stage in practice?
Measure recall@N on a labeled query set: sweep N upward and plot the fraction of queries whose relevant document lands in the pool. Pick the smallest N past the point where recall@N plateaus, since larger N only feeds the reranker more junk and adds latency without raising the ceiling.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Feeding the model the full wide pool of N candidates directly — skipping the reranker — so the prompt fills with low-precision matches and the answer drifts onto distractor chunks.
60 second bullets to scan on the way to the call.
State which symbol names the wide pool and which names the final set
Explain why the first stage optimizes recall and the second optimizes precision
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.