Why are agentic systems shifting toward lazy retrieval via search tools?

Flashcard·Medium·4.0 · 0·~30s·Asked atModal Labs Persistent Salesforce

Attempt it

Eager retrieval fires once with a single query derived from the user's turn, usually a paraphrase. For multi-hop questions ('which papers cited the same theorem as paper X?') one query cannot retrieve everything needed. The agent has to chain queries: retrieve, read, formulate the next query, retrieve again. Lazy retrieval via a search tool lets the model do that chaining inside its reasoning loop. The cost is one extra tool round-trip per retrieval; the benefit is context that grows on demand with what the question actually needs, rather than being guessed at turn start. This is the pattern OpenAI's Deep Research, Claude's research mode, and most coding-agent systems use.

Your answer

Eager retrieval is the classic RAG pattern. At the start of the turn, the system takes the user's question, runs one retrieval, and stuffs the top-k chunks into the context block before the model sees anything. The model never gets to choose what to look up. This works fine when the question can be answered from a single retrieval. The user asks a factual question, the retriever finds the right chunk, the model reads it and answers. It breaks on multi-hop questions where one query is not enough. 'Which papers cited the same theorem as paper X?' requires first finding paper X, then identifying the theorem, then searching for other papers that cited that theorem. A single retrieval at turn start cannot do this, it does not yet know what the theorem is. Lazy retrieval reshapes the problem. Instead of running retrieval before the model thinks, the system exposes search as a tool the model can call during reasoning. The model reads the question, calls search with one query, reads the results, formulates the next query based on what it just learned, and calls search again. Each call is a round trip but the context only grows with what the question actually needs. The trend in 2026 is for any agentic or research-style product to use lazy retrieval. OpenAI's Deep Research, Anthropic's research mode for Claude, Perplexity Pro, and most code agents like Cursor and Windsurf all retrieve iteratively inside the reasoning loop.

Eager retrieval guesses the answer at turn start; lazy retrieval discovers it during reasoning. The shift in 2026 is about which model that better fits the question shape. The eager pattern fires retrieval once before the model sees anything. The query is derived from the user's turn, often a paraphrase or a query-rewrite call, and the resulting top-k goes into the context block as a pre-assembled evidence section. It works for single-hop questions because one targeted query is enough to fetch the answer. The pattern breaks for **multi-hop questions** where the second query depends on the result of the first. 'Which papers cited the same theorem as paper X?' is the canonical example. The system cannot formulate the second query ('papers citing theorem T') without first knowing what T is, which means it cannot be answered at turn start. Eager retrieval responds by paraphrasing the user's question and hoping the right chunks are in the top-k. They usually are not. Lazy retrieval exposes search as a tool the model calls during its reasoning loop. The model reads the question, issues a query, inspects results, formulates a follow-up query, and continues until it has enough to answer. The context grows only with what the model actually needed. The tradeoff is **round trips and tokens**. Each search tool call is one inference round trip and one tool-response payload in context. For a single-hop question this is pure overhead, eager would have been cheaper and faster. For a multi-hop research question it is the only way the system can succeed. The 2026 production pattern is hybrid: a fast eager pass for cheap recall, plus the search tool exposed for the model to refine. OpenAI Deep Research, Claude's research mode, Perplexity Pro, and coding agents (Cursor, Windsurf, Cline) all follow this shape.

TL;DR

Eager retrieval fires one query at turn start and guesses what the question needs; lazy retrieval lets the model issue queries during reasoning so context grows on demand, which is what multi-hop questions actually

Memory aid

Easy to grasp

Imagine asking a research assistant to answer a complicated question. The eager approach is to hand them one folder of articles at the start and say go. The lazy approach is to let them walk to the library, grab a book, read it, decide what to look up next, and walk back for more. For a simple fact-lookup the folder is fine. For a question like which papers cited the same theorem as this one, the assistant needs to walk back to the library several times because each visit tells them what to look for next. Lazy retrieval lets the model do that walking. The cost is a few extra trips; the benefit is finding things one trip could never have found.

Key concepts

Context engineering Retrieval-augmented generation Agents

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Retrieval used to happen exactly once per turn, before the model saw anything. That shape works for single-hop questions and breaks for everything else. The 2026 trend is to let the model decide when retrieval happens, to expose search as a tool the model calls during reasoning. This card walks through what eager retrieval cannot do, what lazy retrieval costs, and the hybrid pattern most production research-style products have settled on.

The eager-retrieval information problem

Eager retrieval runs once at turn start. The system takes the user's question, derives a query (often a paraphrase or an LLM-rewrite), runs the retriever, and concatenates the top-k chunks into the context block. The model only sees the question after the chunks have already been chosen.

This pipeline assumes the right chunks can be predicted from the surface query. For single-hop factual questions that assumption is fine. For multi-hop questions it is not.

Why multi-hop breaks the assumption

A multi-hop question is one whose answer depends on facts that are themselves discovered during reasoning. 'Which papers cited the same theorem as paper X' decomposes into: identify paper X, identify the theorem it relies on, search for other papers citing that theorem. The second-hop query,'papers citing theorem T', cannot be issued at turn start because the system does not yet know T.

Eager pipelines try to work around this with query expansion (paraphrasing the surface question into several variants and retrieving for each). Expansion helps when the right chunk uses different vocabulary than the query, but it does nothing for genuine multi-hop, because the second hop is not a paraphrase of the first.

What lazy retrieval changes

Hybrid eager plus lazy in 2026

Failure modes and bounds

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

search_tool = {
    "name": "search_kb",
    "description": "Search the knowledge base. Returns top-5 chunks with [S<n>] tags.",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string"}},
        "required": ["query"],
    },
}

# Lazy retrieval loop (Anthropic SDK shape)
def run_lazy(user_question: str, max_iterations: int = 8):
    messages = [{"role": "user", "content": user_question}]
    for _ in range(max_iterations):
        resp = client.messages.create(
            model="claude-opus-4-7",
            tools=[search_tool],
            messages=messages,
        )
        if resp.stop_reason != "tool_use":
            return resp.content  # final answer
        for block in resp.content:
            if block.type == "tool_use":
                result = kb.search(block.input["query"])
                messages.append({"role": "assistant", "content": resp.content})
                messages.append({"role": "user", "content": [
                    {"type": "tool_result", "tool_use_id": block.id, "content": result}
                ]})

Real products, models, and research that use this idea.

OpenAI's Deep Research (released 2025, default in ChatGPT Pro 2026) runs an iterative search and read loop with the model calling web search as a tool, sometimes for dozens of rounds.
Anthropic's Claude 4.7 research mode and Projects feature expose file_search and web_search as tools the model invokes during reasoning.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you bound the cost of lazy retrieval in production?

Cap max iterations per turn (8-15 is typical), set per-call token budgets, monitor tool-call distributions, and alert on outliers.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating eager and lazy retrieval as either-or rather than per-workload. Single-hop fact lookups want eager; multi-hop research wants lazy. Many production systems do both.

60 second bullets to scan on the way to the call.

Distinguish eager retrieval (fires before model thinks) from lazy retrieval (model decides when to fire)
Explain why multi-hop questions break eager retrieval

Primary sources. Browse if you want the original framing.