Why does token cost grow super-linearly with agent step count, and what architectural decisions reduce it?

Attempt it

Explain why token cost in an agent loop grows faster than linearly with the number of steps. Describe the asymptotic growth in the worst case, and name three architectural levers that reduce it.

Free · 2 AI evals / day

Quick reference

TL;DR

Each step re-sends the whole growing transcript, so per-step cost rises with step count and total cost is quadratic; pruning, summarisation, and step caps tame it.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine taking notes during a long meeting, but with a strange rule. Before you write each new line, you must first re-read every line you already wrote, out loud, from the very top. Line two costs you one re-read. Line three costs two. By line fifty you are re-reading forty-nine lines before adding anything new. The work to finish the page is not fifty units, it is closer to fifty times fifty divided by two. An agent loop has the same rule. Every turn it must re-read the entire conversation so far before deciding the next move, because the model has no memory between turns. So a longer agent run does not just cost a bit more, it costs dramatically more. Fixes are obvious once you see it. Stop re-reading the old lines, or shrink them into a short summary first.

Key concepts

Cost Compounding Context Management Sliding Window

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

State why statelessness forces a full re-send each turn, derive the triangular sum to show quadratic growth, separate slope-changing levers from the step cap, then add the retry, sub-agent, prefix-caching, and cheaper-model multipliers that show up in real production loops.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's prompt caching bills the unchanged context prefix at a large discount on each agent turn, directly attacking the re-sent token cost of the quadratic.
Claude Code and Cursor run long coding loops that compact older turns into summaries once the transcript nears the context limit, flattening the cost slope on multi-hour runs.
LangGraph exposes state-reducer and trimming hooks so a loop can keep a sliding window of recent messages instead of an append-only transcript.
Multi-agent setups like OpenAI Swarm and LangGraph supervisors route cheap per-step reasoning to a smaller model such as a Haiku-class model, reserving a frontier model like Claude Opus 4.7 for final synthesis.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QPrefix caching bills unchanged context tokens at a discount. Does that change the asymptotic growth, or just the constant?

Sign in or go Pro to see the approach for this follow-up.

QHow does summarisation trade token cost against task accuracy, and how would you tune the compression ratio?

Sign in or go Pro to see the approach for this follow-up.

QIn a multi-agent system, how does total cost scale, and where does coordination overhead enter?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying cost is linear in steps because there are N calls. That counts the calls but ignores that each call re-sends a context that itself grows with the step count.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.