- 1Sort candidates by cross-encoder score and return the top-10 to the consumer
- 2Pass each of the top-100 candidates to a cross-encoder reranker along with the query
- 3Encode the user query with the embedding model
- 4(At index time, separately:) embed every document in the corpus and upsert to the vector DB
- 5ANN search against the embedded corpus to fetch top-100 candidates
Index documents once, then per query: embed query, ANN top-100, cross-encoder rerank, return top-10.
Picture a giant bookshop and a clerk helping you find books. Ahead of time, the clerk has tagged every book with a little label that summarizes what it is about. That tagging is the index-time step. It happens once per book, not once per question. When you walk in with a question, the clerk converts your question into the same kind of label, then quickly grabs a hundred books whose labels look similar. That is the fast first pass. Finally, the clerk reads each of those hundred books a little more carefully alongside your question and picks the ten that actually answer it. That careful reading is the rerank.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Canonical embed then rerank retrieval has five steps split across two timelines, and the order matters because each step depends on the output of the previous one. The question asks you to sequence them correctly. The fastest way to get it right is to notice that one of the steps is parenthetically labeled at index time and the other four are query-time steps; the index-time step has to come first because the others depend on the index existing.
The correct order is [4, 0, 1, 2, 3]: corpus indexing → query embedding → ANN search → cross-encoder rerank → sort and return. The deeper interview probe is whether you can name what each step is doing and why it sits where it does.
The two-timeline split is the architecturally interesting part. Document embeddings are computed once when the corpus is loaded, then reused for every future query. Query embeddings and reranks happen per request. That amortization is exactly why retrieval is fast enough to ship in production: the expensive part runs once, the cheap part runs every time.
The two timelines and why they exist
Step 4, embedding every document in the corpus and upserting to the vector DB, is the only step that runs at index time. The others run at query time, once per user request. The reason this split exists is amortization. Embedding a million documents takes hours or days; doing it once and reusing the vectors across millions of queries makes the per-query cost negligible.
If the corpus embedding step were inside the query-time loop, every query would re-embed every document, which would be wildly impractical. The vector database exists precisely to materialize that work once and serve fast lookups against the cached vectors.
This pattern recurs across the stack. Most expensive operations in retrieval (content-keyed chunk IDs, ANN index builds, learned reranker training) happen once at index or training time and get reused at query time. Confusing the two timelines is one of the most common architectural mistakes in early-stage RAG systems.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Pinecone's reference RAG architecture uses exactly this 5-step shape with OpenAI text-embedding-3-large and Cohere rerank-3.5.
- Anthropic's contextual retrieval recipe pairs the same five-step pipeline with Voyage embeddings and a cross-encoder reranker for production RAG.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat happens if you upgrade the embedding model? Walk through the migration.
Re-embed the entire corpus under the new model, write the new vectors at a separate model_version namespace in the index, validate recall against a labeled set, then atomically flip query traffic to the new model_version. The old vectors stay around until you finish validation, then they get garbage collected. Switching models without re-embedding is a silent corruption.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating corpus embedding as a per-query step. Document embeddings are computed once at index time; only the query embedding is computed per request.
60 second bullets to scan on the way to the call.
List the two timelines (index time vs query time) and which steps live in each.
Explain why corpus embedding cannot move into the query-time path.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.