Why does token cost grow super-linearly with agent step count, and what architectural decisions reduce it?
Explain why token cost in an agent loop grows faster than linearly with the number of steps. Describe the asymptotic growth in the worst case, and name three architectural levers that reduce it.
Each step re-sends the whole growing transcript, so per-step cost rises with step count and total cost is quadratic; pruning, summarisation, and step caps tame it.
Imagine taking notes during a long meeting, but with a strange rule. Before you write each new line, you must first re-read every line you already wrote, out loud, from the very top. Line two costs you one re-read. Line three costs two. By line fifty you are re-reading forty-nine lines before adding anything new. The work to finish the page is not fifty units, it is closer to fifty times fifty divided by two. An agent loop has the same rule. Every turn it must re-read the entire conversation so far before deciding the next move, because the model has no memory between turns. So a longer agent run does not just cost a bit more, it costs dramatically more. Fixes are obvious once you see it. Stop re-reading the old lines, or shrink them into a short summary first.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
State why statelessness forces a full re-send each turn, derive the triangular sum to show quadratic growth, separate slope-changing levers from the step cap, then add the retry, sub-agent, prefix-caching, and cheaper-model multipliers that show up in real production loops.
Real products, models, and research that use this idea.
- Anthropic's prompt caching bills the unchanged context prefix at a large discount on each agent turn, directly attacking the re-sent token cost of the quadratic.
- Claude Code and Cursor run long coding loops that compact older turns into summaries once the transcript nears the context limit, flattening the cost slope on multi-hour runs.
- LangGraph exposes state-reducer and trimming hooks so a loop can keep a sliding window of recent messages instead of an append-only transcript.
- Multi-agent setups like OpenAI Swarm and LangGraph supervisors route cheap per-step reasoning to a smaller model such as a Haiku-class model, reserving a frontier model like Claude Opus 4.7 for final synthesis.
What an interviewer would ask next. Try answering before peeking at the approach.
QPrefix caching bills unchanged context tokens at a discount. Does that change the asymptotic growth, or just the constant?
QHow does summarisation trade token cost against task accuracy, and how would you tune the compression ratio?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying cost is linear in steps because there are N calls. That counts the calls but ignores that each call re-sends a context that itself grows with the step count.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.