Explain why token cost in an agent loop grows faster than linearly with the number of steps. Describe the asymptotic growth in the worst case, and name three architectural levers that reduce it.
Each step re-sends the whole growing transcript, so per-step cost rises with step count and total cost is quadratic; pruning, summarisation, and step caps tame it.
Imagine taking notes during a long meeting, but with a strange rule. Before you write each new line, you must first re-read every line you already wrote, out loud, from the very top. Line two costs you one re-read. Line three costs two. By line fifty you are re-reading forty-nine lines before adding anything new. The work to finish the page is not fifty units, it is closer to fifty times fifty divided by two. An agent loop has the same rule. Every turn it must re-read the entire conversation so far before deciding the next move, because the model has no memory between turns. So a longer agent run does not just cost a bit more, it costs dramatically more. Fixes are obvious once you see it. Stop re-reading the old lines, or shrink them into a short summary first.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent cost compounding is the single most common reason a demo that looked cheap becomes a production system nobody wants to pay for. The cause is not exotic. It falls straight out of one fact: a large language model is stateless between calls. It retains nothing from the previous turn. So the agent runtime has to hand it the entire conversation again on every step, just to re-establish what has already happened.
That re-sending is the whole story. Because the conversation grows every turn, the prompt you pay for grows every turn too. The result is not a model that costs a fixed amount per step. It is a model whose per-step cost climbs as the run gets longer, and a total bill that grows faster than the step count.
The gap between intuition and reality is what trips people up. A naive estimate counts the calls, sees N model invocations, and predicts a cost linear in N. The real bill is closer to N squared, because each of those N calls is itself paying for a slice of context that has been growing the whole time. This section derives that growth precisely, separates the levers that change the growth rate from the ones that merely cap it, and closes on how to measure the curve in a live system before it surprises you on the invoice.
Why statelessness forces a re-send
A chat or agent API call is a pure function of its inputs. The model holds no hidden memory of your previous request. If you want it to act on what happened three turns ago, those three turns must be physically present in the prompt of the current call. The session you see in a chat UI is an illusion maintained by the client, which quietly resends the whole history on your behalf.
The agent runtime maintains this history as a running transcript, often called the context or the scratchpad. It starts as just the user's goal. Each turn appends two things: the model's reasoning and chosen action, then the observation that the executed tool returned. So the transcript grows monotonically, and it never shrinks unless you deliberately make it shrink.
The consequence is that the prompt for step number k contains everything from steps one through k minus one. You are not paying for one step in isolation. You are paying to replay the entire run up to that point, every single turn. This is the structural difference between an agent and a deterministic chain: a chain runs each stage once with a bounded input, while an agent loops over a state that it keeps growing itself.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt caching bills the unchanged context prefix at a large discount on each agent turn, directly attacking the re-sent token cost of the quadratic.
- Claude Code and Cursor run long coding loops that compact older turns into summaries once the transcript nears the context limit, flattening the cost slope on multi-hour runs.
What an interviewer would ask next. Try answering before peeking at the approach.
QPrefix caching bills unchanged context tokens at a discount. Does that change the asymptotic growth, or just the constant?
Work out what fraction of each turn's context is the stable cached prefix versus the new suffix. The sum is still triangular, so the order stays quadratic, but the per-token price on the cached part drops sharply, shrinking the constant. Note caches expire, so a long gap between turns can force a full re-bill.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying cost is linear in steps because there are N calls. That counts the calls but ignores that each call re-sends a context that itself grows with the step count.
60 second bullets to scan on the way to the call.
Explain why statelessness forces the full context to be re-sent each step.
Derive the triangular sum and state why total cost is quadratic in step count.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.