- 1Send the assembled prompt to the model
- 2Keep the top 5 reranked chunks and discard the rest
- 3Restate the user question immediately above the retrieved block
- 4Interleave the top chunks into a head and tail layout with the strongest at top and second strongest at bottom
- 5Score the 50 candidates with a cross-encoder reranker
- 6Embed the user query and run vector search to fetch the top 50 candidate chunks
Retrieve wide, rerank, trim to top-5, then position the survivors to exploit the lost-in-the-middle curve and restate the question right before the block.
Picture packing a tiny suitcase for a long trip. First you pull every plausible item out of the closet, then you sort them by how badly you actually need each one, then you keep only the few that fit. You do not put your passport at the bottom under the shoes. You put it on top where you can grab it, with a backup copy near the zipper for the return flight. Right before you close the case you check the boarding pass one more time so the next thing you do is aligned with the trip. A retrieval pipeline runs the same way. Pull wide, sort by quality, keep the best, place them where the model will actually read them, and remind it what the trip is about.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A retrieval pipeline has six stages, and the temptation is to treat their order as a stylistic choice. It is not. Each stage consumes the output of the previous one, and most reordering errors silently degrade quality without triggering an obvious failure. The pipeline only works as a sequence because each stage is solving a different optimization (recall, precision, budget, attention positioning, anchoring), and those optimizations have a hard dependency graph.
This card walks through why the canonical order is canonical: retrieve, rerank, trim, position, restate, send. We will also look at where the lost-in-the-middle curve enters the picture and why the question restate sits where it does.
Stages 1-2: recall then precision
The first two stages form a classic recall then precision handoff. Stage 1 is wide retrieval. You embed the user query and pull top-30 to top-50 candidates from a dense or hybrid index. The objective is purely recall: is the right chunk somewhere in the candidate set? Cost scales linearly with index size, which is why this stage uses a bi-encoder with precomputed chunk embeddings and an ANN lookup.
Stage 2 is rerank. A cross-encoder scores each (query, chunk) pair jointly. Joint scoring lets attention compare every query token against every chunk token, which produces a sharper signal than the cosine of two independent embeddings. The cost is one inference per candidate, so stage 2 only makes sense over the small set produced by stage 1.
Why this dependency is hard
Reranking has nothing to score before stage 1 produces candidates. Running stage 1 over a wider window does not help stage 2 if stage 2 is bypassed. The two stages have to run in order and both have to run.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude Opus 4.7 RAG cookbook recommends wide dense retrieval, cross-encoder rerank, top-5 trim, and explicit restate of the user query above the block.
- Cohere's 2026 production RAG guide pairs Embed v4 with Rerank 3.5 and ships a positioning helper that interleaves chunks for head and tail layout.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere would you place a system prompt with hard rules relative to the retrieved block?
System prompts go at the very top of the context, before retrieval, so they sit at the head of the lost-in-the-middle curve and can be prompt-cached across turns.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reranking before retrieval is impossible; trimming before reranking wastes the precision step; positioning before trimming positions garbage. Stage order is not stylistic, it is causal.
60 second bullets to scan on the way to the call.
Order: retrieve, rerank, trim, position, restate, send
Why retrieve has to come before rerank
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.