Each agent turn re-prefills the whole transcript then waits on the tool, so latency compounds with turn count; parallel calls collapse the dominant tool-time axis.
Imagine asking a research assistant a question that needs three library books. A slow assistant fetches one book, rereads everything written so far, walks back, fetches the next, rereads everything again, and repeats. Most of the wasted time is the walking, not the reading. If the three books are unrelated, you could send the assistant to grab all three in one trip instead of three. That single change saves far more time than asking them to read a little faster. Each round trip also forces a reread of everything so far, so the more trips you make, the worse it gets. Fewer, parallel trips beat a faster reader almost every time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent latency is one of the most misdiagnosed problems in production LLM systems. Engineers reach instinctively for the levers they know from single-request serving: a faster GPU, a smaller model, FP8 KV cache, a higher batch size. Those levers target the model forward pass. But an agent's wall-clock time is rarely dominated by the forward pass. It is dominated by the structure of the loop around it.
An agent loop is a sequence of turns. On each turn the model reads the entire transcript so far, decodes an action (often a tool call), the runtime executes that tool, and the result is appended to the transcript before the next turn begins. Three costs stack on every turn: prefill of the growing transcript, decode of the response, and the external tool plus network time. The total latency a user feels is the sum of all of these across every turn, and that sum is what a product latency budget actually has to fit inside.
The reason this trips people up is that the model itself feels like the expensive, exotic component, so attention goes there by default. In practice a single decode of a short tool call is cheap, while the database read or third-party API behind the tool is slow and entirely outside the model's control. The mental shift a senior engineer makes is to stop thinking about latency as a property of the model and start thinking about it as a property of the loop: how many sequential round trips, how big the transcript grows, and how much of the prefill repeats.
This deep dive builds the latency model term by term, shows why tool time and turn count dominate, walks through why each MCQ distractor is the wrong axis, and then derives the two highest-leverage mitigations: parallel tool calls, which collapse the tool axis when calls are independent, and prompt caching, which collapses the compounding prefill axis. By the end you should be able to look at an agent trace and say exactly which term to attack and roughly how much you stand to gain.
Anatomy of one agent turn
Strip an agent down to a single turn and three distinct costs appear. First, prefill: the model must process the entire prompt, which is the system prompt, the tool definitions, and the full transcript of prior turns. This is a parallel, compute-bound pass, but its cost grows with the length of everything that came before. On a long-running agent the prefill of turn ten can be many times the prefill of turn one, purely because the transcript has accumulated.
Second, decode: the model generates its response token by token. This is the autoregressive, memory-bandwidth-bound phase that single-request serving optimizations target. For a short tool call, decode is often only a few dozen tokens, so even at a modest tokens-per-second rate it finishes in well under a second. Decode only becomes the dominant term when the model emits long free-text answers, not when it emits compact tool calls.
Third, tool execution: the runtime takes the emitted tool call, makes a network round trip to wherever the tool lives, the tool does its work, and the result comes back. A database query, a vector search, or a third-party API can take anywhere from tens of milliseconds to several seconds. This term has nothing to do with the model at all, yet it frequently dwarfs both prefill and decode.
The practical consequence is that the per-turn cost is lopsided. Two of the three components live inside your serving stack and are well optimized; the third lives across a network boundary and is frequently the slowest. Any honest latency analysis has to measure all three separately rather than assuming the model is the expensive part.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude Opus 4.7 tool-use API returns multiple tool_use blocks in one turn so a client runtime can execute independent calls concurrently.
- OpenAI's function-calling and the GPT-5.5 Responses API emit parallel tool calls in a single assistant message for independent lookups.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does prompt caching help an agent loop more than a single one-shot completion?
In a loop the stable prefix (system prompt, tool definitions, early transcript) is re-sent every turn. Caching it means each turn re-prefills only the new suffix, turning roughly quadratic total prefill work into linear across the loop.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Optimizing per-token decode speed (FP8 cache, smaller model) while ignoring that tool execution and network round trips, multiplied by the number of turns, dominate the wall clock.
60 second bullets to scan on the way to the call.
Why latency compounds with the number of turns in an agent loop
What re-prefilling the growing transcript costs on every turn
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.