Sizing a retrieve then rerank funnel: how do you pick N and the final k?
A two stage RAG retriever fetches N candidates with a cheap vector search, then a cross-encoder reranks them down to the k chunks sent to the LLM. Explain what N and k each control, and how you would tune them.
N is how wide the cheap retriever casts for recall; k is how many reranked chunks reach the prompt. Tune N up until recall@N plateaus, then pick the smallest k that holds answer quality.
Imagine hiring for one job. First a quick resume filter pulls fifty plausible applicants from thousands — fast and cheap, but rough, so you grab a wide pile to be sure the best person is in it. That pile size is N. Then a careful interviewer reads all fifty closely and ranks them properly; that careful read is too slow to run on thousands, which is why the quick filter goes first. Finally you make offers to the top few, say five. That final number is k. If your first pile is too small, the best candidate never gets interviewed. If you bring too many to the final round, you waste time and start confusing good candidates with mediocre ones. Picking N and k well is just balancing those two squeezes.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This is a sizing question, and sizing questions separate people who have run a RAG eval from people who have only drawn the architecture diagram. N and k are two numbers in the same pipeline, but they control completely different failure modes, and the tells of a weak answer are treating them as interchangeable or tuning them by gut.
The deeper point the question is probing is the cost asymmetry between the two stages. A vector search is cheap and runs over the whole index; a cross-encoder is expensive and can only afford to run on a short list. Everything about how you choose N and k flows from that asymmetry. N is bounded by how much reranking you can afford and by where recall stops improving; k is bounded by how much context the generator can use before extra chunks start hurting. A strong answer makes both bounds concrete and ties each to a number you can actually measure on an eval set.
Why two stages instead of one
A single-stage retriever has to choose between cheap and accurate. A bi-encoder vector search is cheap: it embeds query and documents independently and ranks by a dot product, so it scans the whole index fast via an ANN structure. But independent embeddings miss fine-grained query-document interactions, so the ordering near the top is rough.
A cross-encoder is accurate: it concatenates the query with a candidate and encodes them jointly, letting query tokens attend to document tokens, which captures matches the dot product cannot. The price is that it must run the model once per query-document pair. Running it across a million-document corpus per query is hopeless on any latency budget.
The funnel resolves the tension. The cheap stage scans everything and proposes N candidates with good recall but rough ordering. The expensive stage then runs only on those N and produces a precise ordering. You get corpus-wide recall and cross-encoder precision without paying cross-encoder cost over the corpus. N is the seam between the two stages, and k is what leaves the funnel for the prompt.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Cohere Rerank is a hosted cross-encoder that teams drop in as the second stage, reranking the top N vector hits down to a small k.
- LlamaIndex and LangChain expose a similarity_top_k for the retriever (N) and a separate reranker top_n (k) as two distinct knobs in the query pipeline.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide between a fixed k and a dynamic relevance threshold from the reranker scores?
A fixed k is simple but wastes context on easy queries and starves hard ones; a score floor adapts k per query, at the cost of needing well-behaved, comparable reranker scores.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Setting N too small, so the gold chunk never enters the pool — the reranker can only reorder what it is handed, it cannot recover a missed chunk.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.