Which mitigation has the largest impact on agent-loop latency when tool calls are independent?

An agent loop is a cycle. The model emits a tool call, your runtime executes the tool, the result is appended to the transcript, and the model runs again. Each pass through that cycle has its own cost. The key thing to see is what happens on every turn. The model must re-read the entire growing transcript before it can decode the next action. That re-reading is the **prefill** phase, and it gets longer every turn because the transcript keeps growing. Then the model decodes its response, and then you pay the external tool time: a network round trip plus however long the tool itself takes. Latency compounds with the number of turns. A five-turn agent loop pays five prefills, five decodes, and five tool round trips, stacked end to end. The slow part is almost always the tool time, not the model. In the question, three options tweak small or unrelated knobs. FP8 cache helps decode bandwidth. Temperature changes sampling, which is microseconds. More tokens makes things slower, not faster. The winning option attacks the dominant axis. If several tool calls do not depend on each other, issue them in parallel inside one turn. That collapses several sequential tool round trips into roughly the time of the single slowest call.

Agent latency is governed by the loop, not the single forward pass. Each turn re-prefills the entire growing transcript, decodes an action, then blocks on an external tool. The dominant term is tool plus network time, multiplied by the number of turns. A database query or external API can take hundreds of milliseconds to seconds. Decode is often the smaller cost. So the highest-leverage move is to cut either the number of sequential turns or the per-turn tool wait. Parallel tool calls do exactly that. When calls are independent, the model emits them in one assistant turn and the runtime fires them concurrently. The sequential cost of running calls one after another collapses to roughly the slowest single call. That is a near N-fold cut in the tool dimension for a fan-out of N. The distractors miss the dominant axis. FP8 KV cache speeds per-step decode, real but secondary here. Lower temperature only changes sampling, which is microseconds. Raising the token budget makes a turn longer, not shorter. The second lever is prompt caching. It avoids re-prefilling the stable prefix every turn. That directly attacks the compounding cost of the regrowing transcript.