Three layered caps stop a runaway agent loop: a step budget on tool calls, a token budget per request, and a per-trace USD ceiling at the gateway.
Picture giving a junior employee a company credit card and telling them to keep researching until the answer is good. Without limits, they might call thirty experts, print a thousand pages, and rack up a giant bill. Three rules keep them safe: a cap on how many phone calls they can make, a cap on how much paper they can print, and a hard dollar ceiling on the card itself. Each rule catches a different runaway pattern. One call that burns a fortune. Many tiny calls that compound. A clever vendor who somehow gets around both. An agent loop is the same idea. The step cap, the token cap, and the dollar cap each plug a different leak.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent loops are the new cost-incident shape that LLMOps had to learn the hard way. A classical web service caps cost implicitly: each request runs one handler, the handler returns, the database query has a timeout. An agent has none of that structure by default. The LLM plans a step, calls a tool, reads the result, decides what to do next. Nothing in the loop guarantees it ever exits. If the model gets confused, or the task description is open-ended, or a prompt-injection payload in a tool result tells the model to keep calling tools, the loop runs until something else stops it.
What 'something else' should be is the design question. Production teams converge on three layered caps, each enforced at a different layer of the stack, each catching a different failure mode. This walkthrough breaks down what each cap stops, where it lives, and why removing any one leaves a real exposure.
Operating principle: treat agent runaways as a security and cost concern simultaneously. Prompt injection is the most common deliberate trigger; ambiguous task framing is the most common accidental one. Both look identical at runtime, and the same caps stop both.
Cap 1: step budget in the runtime
What it does
A step budget limits the number of tool-call iterations one request can run. LangGraph calls it recursion_limit, CrewAI calls it max_iter, the OpenAI Agents SDK calls it max turns. The runtime tracks iteration count and forces the loop to exit when the count crosses the threshold, returning whatever partial answer the agent has produced so far.
Why it is necessary
It stops the infinite loop. A model that keeps calling the same tool because it cannot interpret the result, or keeps planning more steps because the task is open-ended, would otherwise loop forever or until provider rate limits intervene. Rate limits are a poor cost control because they trigger only after thousands of calls.
Why it is not sufficient
A single tool call can be expensive. A web-fetch tool that returns a 200K-token document, an internal-search tool that returns 50 results with full bodies, a code-execution tool that returns a megabyte of stdout. The step budget treats each as one tick; the gateway sees one iteration but huge tokens. The token budget catches this.
How to set it
Per route, not globally. A research agent that genuinely needs multi-step reasoning gets 25. A customer-support reply agent that should fetch one record and answer gets 5. Hitting the cap should emit a structured event so SRE can alert on the rate of budget breaches, not just the financial consequence.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangGraph exposes `recursion_limit` on every graph; the default is conservative because runaway loops are a known failure class.
- LiteLLM and Portkey both expose per-request token and cost caps as gateway-level guardrails enforceable across providers.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect a slow-cooking agent runaway before the cost ceiling fires?
Track running cost per trace as a span attribute streamed to your observability backend. Alert when p99 over a 5-minute window crosses 3x the rolling 7-day baseline. Surfaces the trend before any single trace breaches the hard ceiling.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Relying on a single cap (usually step count) and assuming it covers cost. A single tool call that returns a 200K-token document blows past the token budget on iteration one.
60 second bullets to scan on the way to the call.
The three cap layers and which failure mode each prevents
Where each cap is enforced (runtime vs gateway) and why
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.