Why does an agent loop with 5 tool calls have ~6× the latency of a single completion and how do you mitigate?
An agent loop performs 5 sequential tool calls before final answer. Walk through where the latency comes from and explain why total wall clock is ~6× a single completion. Identify three mitigations and what each costs.
An agent loop chains 6 model turns plus 5 tool round-trips, and context grows every turn, so prefill, decode, and network all stack into roughly 6x a single completion.
Imagine cooking a recipe where you must phone a friend after every step to ask what comes next. Each call has the same overhead: you read the whole recipe so far aloud, your friend thinks, you wait on the line, they answer, then you act. As the recipe grows, reading it aloud each time takes longer and longer. Five questions means six phone calls plus five waits for your friend to act, and every call re-reads more text than the last. The cooking itself is fast. The phone tag is what eats the clock. You speed it up by asking several independent questions in one call, by not re-reading the parts your friend already heard, and by starting the next step before they finish replying when you are confident.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent loop latency is one of the most reliable senior interview filters in inference optimization, because it forces the candidate to reason about a multi-turn system rather than a single forward pass. A single completion is one prefill plus one decode. An agent that calls five tools before answering is a fundamentally different beast: six model turns interleaved with five tool round-trips, where the context the model must read grows on every turn.
The headline number, roughly six times a single completion, is correct as a first approximation. But the interesting part is why it is not exactly six. The turns are not equal. Each tool result is appended to the conversation and never removed, so the prompt the model prefills at turn six is far longer than at turn one. Prefill cost rises with context length, which makes the total super-linear in the number of turns.
This deep dive decomposes a single tool-call round into its components, explains why prefill is the term that grows, and then maps each of the three standard mitigations onto the specific latency axis it attacks. By the end you should be able to walk an interviewer through the full timeline and justify which mitigation to reach for in a given product.
Decomposing a single tool-call round
One round of an agent loop is a chain of distinct phases, and naming each one is what separates a strong answer from a vague one. The phases are decode to the tool stop, serialize the call, network out to the tool, tool execution, network back, prefill of the grown context, and finally decode of the next assistant turn.
Decode to the tool stop is the model generating tokens autoregressively until it emits a complete tool call. For a short tool message this is one to three seconds, dominated by per-token decode latency. Serialization and the two network hops are usually small but real, on the order of tens to low hundreds of milliseconds each depending on where the tool runs.
Tool execution itself is wildly variable. A local function returns in microseconds. A web search or a database query can take seconds. Then the result is appended to the conversation and the next turn begins with a prefill of the entire accumulated context. That prefill is the term most candidates forget, and it is the one that grows.
The reason the round multiplies into a 6x figure is that none of these phases overlap by default. The model cannot start the next decode until the tool result comes back, the tool cannot start until the model finishes decoding the call, and the prefill cannot start until the result is appended. Each phase blocks the next. So a 5-tool loop serializes six decodes, five prefills, and five tool executions end to end, and a strong answer narrates exactly that timeline before reaching for any fix.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude Opus 4.7 supports parallel tool calls in a single turn plus prompt caching that bills cached prefix tokens at roughly a tenth of normal input cost.
- OpenAI's GPT-5.5 function calling returns multiple tool calls per turn, letting the agent runtime fan them out concurrently.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does prefill grow super-linearly across an agent loop rather than staying constant?
Each tool result is appended permanently, so the prompt at turn k contains all prior results. Prefill is roughly linear in context length, so the per-turn prefill cost rises every turn and the total is the sum of a growing series, not a flat multiple.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming tool execution or the network for the latency while ignoring that the growing context forces a larger prefill on every single turn of the loop.
60 second bullets to scan on the way to the call.
Counting turns and round-trips for an N tool agent loop correctly
The full component breakdown of one tool-call round
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.