Design an iteration aware token budget for an agent that may run up to 50 steps

Short answer·Hard·4.0 · 0·~3 min·Asked atReplicate Sharechat Stripe

Attempt it

An agent has a 128k window and may run up to 50 iterations. Design a token budget policy that does not run out of room halfway through a long run. Cover the static prefix, summary growth, scratchpad, and tool results.

Free · 2 AI evals / day

A long-running agent does not get to spend its **token budget freely**. A 128k window sounds enormous, but across 50 iterations the scratchpad, tool results, and message history compound fast. The fix is to carve the window into named slots and govern each one. Start with the **static prefix**. That is the system prompt plus tool definitions. Cap it at around 4k and keep its bytes identical across iterations so prompt caching works. Next, reserve a small slot (about 2k) for persistent memory, facts the agent must always see, like the user goal or a constraint. Reserve another 8k at the top for the final output. That leaves around 114k of working area. Inside the working area, run a **hierarchical summary**. Every five iterations, compress that block into one level-one summary. Every five level-one summaries, compress them into a level-two summary. The most recent three to five tool results live verbatim because the agent is actively reasoning over them. Older results fold into the running summary. Cap each tool result at around 4k. If a tool returns more, save the overflow out of band and replace it with a reference like `[stored at result_id=...]`. Set a soft watermark at 80 percent of the window to trigger a compaction pass, and a hard watermark at 95 percent that forces aggressive eviction. The result is a budget that stays roughly flat across all 50 iterations.

An iteration-aware **token budget** for a 50-step, 128k-window agent must keep the prompt size roughly flat across the trajectory rather than growing with iteration count. The design has four moving parts. **Fixed slots first.** Cap the static prefix (system prompt and tool definitions) at 4k and keep it byte-stable so prompt caching covers it. Reserve 2k for persistent memory (the user goal, hard constraints, key entities). Reserve 8k of headroom for the final output. That leaves roughly 114k as the working area. **Hierarchical summarization in the working area.** Recent iterations live verbatim. Every five iterations, the oldest block compresses into a level-one summary of around 500 tokens. Every five level-one summaries compress further into a level-two summary. This bounded recursion is the only way the agent can recall what happened in iteration three while running iteration forty-five without consuming proportional tokens. **Tool result discipline.** Cap each tool result at 4k. Anything larger is written to an out of band store and replaced inline with a reference (`{result_id, summary, schema}`). The agent can `read_full_result` on demand. The three to five most recent results stay verbatim; older ones fold into the level-one summary. **Watermark-driven compaction.** Track current tokens against the working area. At a **soft watermark of 80 percent**, run a compaction pass: collapse the oldest verbatim turns into the active level-one summary and re-emit the scratchpad. At a **hard watermark of 95 percent**, evict aggressively: roll oldest level-one summaries into a level-two summary, drop intermediate tool results entirely, keep only references. The property this design enforces is that prompt size has a ceiling determined by the slot sizes, not by iteration count. Whether the agent is on step five or step forty-five, the prompt fits and the next call has predictable cost and latency.

TL;DR

Carve the window into a small static prefix, a small persistent memory slot, a reserved output slot, and a large working area governed by a hierarchical summary plus soft and hard watermarks that trigger compaction

Memory aid

Easy to grasp

Think of your suitcase before a 50-day trip. You cannot bring 50 fresh outfits, it will not fit. So you set aside one pocket for documents (the static prefix, never changes), one pocket for essentials (persistent memory), and you keep one corner empty for the souvenir you will pack last (the output). The rest of the suitcase is for clothes, and you cycle them: every five days you bundle the dirty ones into one compact pouch (a level-one summary). When the suitcase looks 80 percent full, you re-roll the pouches. When it looks 95 percent full, you bundle the oldest pouches together into one even smaller bundle. The shape never changes, so you always have room for tomorrow.

Key concepts

Context Engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Designing an iteration-aware token budget is what separates demo agents from production agents. A 128k window across 50 iterations sounds generous on paper, but without explicit budget discipline the scratchpad, the tool results, and the message history will combine to overflow somewhere between iteration 20 and iteration 35. The fix is not a bigger model, it is a budget design that keeps the prompt roughly flat in size across the trajectory. This deep dive walks through the four-slot layout, the hierarchical summary that governs the working area, the tool-result discipline, and the watermark policy that ties it all together.

Why a flat window is not enough

A naive long-running agent appends every model reply, every tool call, and every tool result to a growing message list. Over 50 iterations even modest per-turn additions compound: 2k of scratchpad plus 3k of tool result per turn is 250k after 50 turns, which is roughly twice the 128k window.

The agent does not crash gracefully. What happens is that around iteration 25 the prompt approaches the window, the provider truncates the oldest messages (silently in some APIs, with an error in others), and from that point the agent has forgotten its goal, its plan, and its earlier tool results. The trajectory continues but the work is wasted.

The insight that fixes this is that the window is not the budget. The budget is what is left after you reserve room for the things the agent always needs: the system prompt, the persistent memory, and the output. The working area is everything else, and it must be governed by an eviction policy that runs on every iteration, not just when overflow is imminent.

The four-slot layout

Hierarchical summary in the working area

Tool result discipline

Watermark policy and eviction order

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's Claude Opus 4.7 agent loops in production use exactly this slot plus watermark discipline for multi-hour code agents
LangGraph's MessagesState reducer combined with a summarization node implements the hierarchical summary pattern directly

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you tune the watermark thresholds for a latency-sensitive vs a cost-sensitive agent?

Latency-sensitive agents want compaction to happen rarely because a compaction pass is an extra LLM call. Raise the soft watermark to 90 percent and let evictions happen more often. Cost-sensitive agents want the prompt to stay small, so lower the soft watermark to 70 percent and accept the extra compaction cost. Replay representative trajectories to find the sweet spot for the workload.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating the 128k window as one undifferentiated pool, then watching the agent crash at iteration 30 when the unbounded scratchpad and uncapped tool results have eaten the whole budget.

60 second bullets to scan on the way to the call.

Name the four slots and the approximate size of each
Explain why the static prefix must stay byte-stable

Primary sources. Browse if you want the original framing.