Why does an agent loop with 5 tool calls have ~6× the latency of a single completion and how do you mitigate?

An **agent loop** is not one model call. It is a back and forth. The model decodes a tool request, the runtime executes that tool, the result is appended to the conversation, and the model is called again. Five tool calls means six model turns and five tool round-trips before the final answer. Walk through one round. The model decodes tokens until it emits a tool call, which takes a second or two. The call is serialized and sent over the network to the tool. The tool runs and returns a result. That result is appended to the context, and the next model turn must prefill the entire grown context before it can decode anything new. The expensive part hides in that prefill. Each turn the prompt is longer, because every tool result stays in the context. So turn 6 has to read far more tokens than turn 1 did. Latency does not just add up linearly. The prefill cost climbs turn over turn. The fixes attack different pieces. Run independent tools at the same time. Cache the unchanging prefix so you stop re-reading it. Start a slow tool early when you are confident about what comes next.

**An agent loop with 5 tool calls is 6 model turns plus 5 tool round-trips.** Each round is the same chain. The model decodes until it emits a tool call. The call is serialized and sent over the network. The tool executes. The result returns. The next turn prefills the grown context, then decodes again. The hidden cost is prefill. Every tool result stays in context, so the prompt grows monotonically. By turn 6 the prefill may read two to three times the tokens of turn 1. Latency compounds rather than just adding. Three mitigations each target a different component. Parallel tool calls let the model request several independent tools in one turn. The runtime fires them concurrently, collapsing five round-trips toward one. The cost is that dependent tasks cannot parallelize. Prompt caching keeps the stable prefix warm across turns. It cuts input cost to roughly a tenth and skips re-prefilling the cached span, lowering time to first token. The cost is a careful, stable prefix layout. Speculative execution starts an expensive tool before the request is fully confirmed. It hides tool latency when the guess is right. The cost is wasted work on mispredictions. Production agents combine all three.