Predict the retrieval trace: how a multi-hop RAG agent answers a chained question that single shot RAG would fail on.
A multi-hop RAG agent is given the question: 'What was the revenue impact in fiscal year 2025 of the cloud product launched by the CEO who replaced the founder of Stripe?' The agent runs an iterative loop where each step issues one retrieval query, reads the chunks, and decides the next query (Self-Ask / IRCoT style). Assume the corpus contains the relevant Wikipedia and SEC filing chunks. Write the sequence of retrieval queries the agent issues, in order, ending when it has enough to answer. For each query, in one phrase say what fact it surfaces.
The question chains three facts: who replaced the founder, what that person launched, and the revenue impact. Each hop depends on the previous answer, so the agent issues three sequential queries.
Imagine someone asks you who painted the ceiling of the chapel where Pope Julius II is buried. You cannot answer that in one library trip. First you find out where Pope Julius II is buried, then you find out who painted the ceiling of that chapel. Two trips, and the second trip needs the answer from the first. Multi-hop RAG works the same way. Each step asks one sub-question, looks at the answer, and uses it to write the next sub-question. A single-shot search would not work because no single book has the full chain in one place. The agent strings together separate facts by doing one careful lookup at a time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-hop RAG is the standard answer to questions that chain multiple facts together. The example in the prompt chains three: founder successor, product launch, revenue impact. No single chunk in any realistic corpus will contain all three, so single-shot RAG structurally cannot answer it.
What the senior interviewer wants to see is whether you can articulate the dependency between hops, walk the trace correctly, and explain why this is a different problem from multi-query retrieval. This deep dive walks the retrieve and reason loop, names the canonical papers (Self-Ask, IRCoT, Adaptive RAG), and closes with the production realities: latency multiplier, error propagation, and the gating decision.
Why single-shot RAG fails on chained questions
A single-shot RAG pipeline embeds the user question once, retrieves the top-k chunks, and passes them to the generator. Embedding similarity is topical: a chunk is retrieved because it is semantically near the question. For a chained question like the one in the prompt, the topical neighborhood is wide: chunks about Stripe, chunks about CEO successions, chunks about cloud products, chunks about FY2025 financial impact. Even with top-k = 20, the chance that all three required facts land in the same retrieved set is low.
The deeper structural reason is that the chain itself does not exist in the corpus. Wikipedia has an article about Stripe leadership. SEC filings have a section about a specific product's revenue. A press release announces a product launch. These three chunks live in three different documents written for three different purposes. The chain that connects them lives only in the user's question.
Multi-hop reconstructs the chain by traversing it explicitly. Each hop produces an intermediate fact, and the next hop's query is conditioned on that fact. The agent is doing what a human researcher would do: open one tab, find a name, open a second tab with that name, find a product, open a third tab with that product, find the number.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Sketch of a multi-hop retrieve and reason loop, IRCoT-style.
from anthropic import Anthropic
client = Anthropic()
async def multihop_answer(question: str, max_hops: int = 5) -> tuple[str, list[dict]]:
facts: list[dict] = [] # accumulated (sub_q, chunk, extracted_fact)
for hop in range(max_hops):
# 1. Decide the next sub-question given what we already know.
plan = await client.messages.create(
model="claude-opus-4-7",
messages=[{"role": "user", "content": plan_prompt(question, facts)}],
)
sub_q = parse_subquestion(plan)
if sub_q is None:
break # planner says we have enough
# 2. Retrieve for this sub-question.
chunk = await retrieve_top_chunk(sub_q)
# 3. Extract the fact relevant to the chain.
extracted = await extract_fact(sub_q, chunk)
facts.append({"sub_q": sub_q, "chunk": chunk, "fact": extracted})
# 4. Synthesize final answer with citations.
return await synthesize(question, facts)| Aspect | Single-shot RAG | Multi-hop RAG |
|---|---|---|
| Queries per question | 1 | k (typically 2-5) |
| Latency | ~1.5 s P95 | ~k × 1.5 s P95 |
| LLM calls | 1 | k+1 (one per hop + synthesis) |
| Cost multiplier | 1x | 2-5x |
| Strength | single-fact questions | chained / comparative questions |
| Failure mode | misses chain facts | error propagates across hops |
Real products, models, and research that use this idea.
- LangChain Self-Ask with Search agent runs this exact retrieve and reason loop, using a follow-up question marker to drive each hop.
- DSPy's IRCoT module implements the IRCoT pattern, interleaving chain-of-thought reasoning with retrieval at each step.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the agent know when to stop the loop?
A planner LLM call at the top of each iteration decides 'do I have enough facts to answer?' and outputs either the next sub-question or a stop token. Confidence calibration matters; over-eager stops produce incomplete answers, under-eager stops burn budget.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Issuing one query against the raw chained question. A single search cannot surface three distinct facts at once if no chunk contains the full chain.
60 second bullets to scan on the way to the call.
Why the question's three facts cannot be served by one retrieval
How each hop's query depends on the previous answer
Primary sources. Browse if you want the original framing.
- Press, Ofir et al., Measuring and Narrowing the Compositionality Gap in Language Models (Self-Ask)
- Trivedi et al., Interleaving Retrieval with Chain-of-Thought Reasoning (IRCoT)
- Jeong et al., Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop QA
Same topic, related formats. Practice these next.