Spot the errors in this agent loop's thinking budget configuration
Click any words you think contain an error. Click again to unmark.
Three bugs: uniform max budget across all steps, a wrong claim that thinking is cached and free, and replaying every prior reasoning trace into the next turn's input.
Picture a project manager who tells a team of consultants to spend a full day deeply researching every single task, from a one-line status update to a board strategy memo. That is the first bug, same effort budget for every job, big or small. Then imagine they tell finance the consultants work for free because someone in the cafeteria heard a rumour. Wrong, they bill by the hour, and at a high rate. Finally imagine the PM forwards every consultant's scratch notes to every other consultant before every new task, choking everyone's inbox with stale paper that gets thrown away on arrival. A working agent budget varies effort by step, accepts that thinking is billed output, and only replays the thinking the next turn actually needs.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent loops are where reasoning-model costs go feral. A single chat call with a thinking budget is bounded by one user turn. An agent runs 5 to 50 steps per task, each with its own thinking budget, each billed at the output rate. Without discipline, the same agent that worked beautifully in a notebook turns into a five-figure monthly line item the first week it hits production.
The config in front of us captures the three classic mistakes a team makes on their first reasoning-agent deploy. This deep dive walks through each error, the precise fix, and the operational pattern that makes a reasoning agent cost-predictable at scale.
Bug one: a flat budget pretends every step is the same
Inside an agent loop, step difficulty spans an order of magnitude or more. A single agent run might include:
- Format a JSON tool argument from one observation. Zero reasoning useful.
- Pick which of five tools to call. Light reasoning helps.
- Decide whether the last three tool results refuted the working hypothesis. Real deliberation pays off.
- Synthesise the final answer from accumulated evidence. Maximum effort warranted.
A flat 32k ceiling on all 20 steps charges the maximum on every one. The model does not 'use less when it does not need it', because the configured budget is the soft target and the trace tends to expand to fill it.
The fix: a per-step effort policy
The right pattern is a small effort tier set, off, low, medium, high, plus a cheap router that picks the tier for each step. The router runs on rules first, with a fast classifier as fallback. Inputs to the router are step type (planner, tool-call, observation-process, synthesiser), prompt length, recent failure signals, and the agent's running progress estimate.
A realistic production distribution on a mixed workload is more than 80 percent of steps at off or low, with high reserved for the few planning and synthesis steps that actually move the needle. The total reasoning budget per run drops by 10x or more with no measurable quality loss because the dropped tokens were never doing work.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic Claude Opus 4.7 extended thinking documentation explicitly states thinking tokens bill at the standard output rate and are not served from prompt cache
- OpenAI o-series API strips reasoning items from prior assistant turns on the next request, with the Responses API offering an opt-in continuation pattern for the same turn only
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build the per-step effort router cheaply enough that it does not eat the savings it creates?
Start with a rules table keyed on step type and observation length, escalate to a tiny embedding-based classifier only when rules saturate. The router itself should run on a fast non-reasoning model or pure logic, with sub-millisecond decision latency.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating thinking budget as a single knob set once for the whole loop, instead of a per-step decision tied to that step's difficulty and value.
60 second bullets to scan on the way to the call.
How agent step difficulty varies and what features a step-classifier can use
Why uniform per-step budgets multiply cost without buying quality
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.