An agent has a 128k window and may run up to 50 iterations. Design a token-budget policy that does not run out of room halfway through a long run. Cover the static prefix, summary growth, scratchpad, and tool results.
Free · 2 AI evals / day
A long-running agent does not get to spend its **token budget freely**. A 128k window sounds enormous, but across 50 iterations the scratchpad, tool results, and message history compound fast. The fix is to carve the window into named slots and govern each one.
Start with the **static prefix**. That is the system prompt plus tool definitions. Cap it at around 4k and keep its bytes identical across iterations so prompt caching works. Next, reserve a small slot (about 2k) for persistent memory, facts the agent must always see, like the user goal or a constraint. Reserve another 8k at the top for the final output. That leaves around 114k of working area.
Inside the working area, run a **hierarchical summary**. Every five iterations, compress that block into one level-one summary. Every five level-one summaries, compress them into a level-two summary. The most recent three to five tool results live verbatim because the agent is actively reasoning over them. Older results fold into the running summary.
Cap each tool result at around 4k. If a tool returns more, save the overflow out of band and replace it with a reference like `[stored at result_id=...]`. Set a soft watermark at 80 percent of the window to trigger a compaction pass, and a hard watermark at 95 percent that forces aggressive eviction. The result is a budget that stays roughly flat across all 50 iterations.
An iteration-aware **token budget** for a 50-step, 128k-window agent must keep the prompt size roughly flat across the trajectory rather than growing with iteration count. The design has four moving parts.
**Fixed slots first.** Cap the static prefix (system prompt and tool definitions) at 4k and keep it byte-stable so prompt caching covers it. Reserve 2k for persistent memory (the user goal, hard constraints, key entities). Reserve 8k of headroom for the final output. That leaves roughly 114k as the working area.
**Hierarchical summarization in the working area.** Recent iterations live verbatim. Every five iterations, the oldest block compresses into a level-one summary of around 500 tokens. Every five level-one summaries compress further into a level-two summary. This bounded recursion is the only way the agent can recall what happened in iteration three while running iteration forty-five without consuming proportional tokens.
**Tool result discipline.** Cap each tool result at 4k. Anything larger is written to an out of band store and replaced inline with a reference (`{result_id, summary, schema}`). The agent can `read_full_result` on demand. The three to five most recent results stay verbatim; older ones fold into the level-one summary.
**Watermark-driven compaction.** Track current tokens against the working area. At a **soft watermark of 80 percent**, run a compaction pass: collapse the oldest verbatim turns into the active level-one summary and re-emit the scratchpad. At a **hard watermark of 95 percent**, evict aggressively: roll oldest level-one summaries into a level-two summary, drop intermediate tool results entirely, keep only references.
The property this design enforces is that prompt size has a ceiling determined by the slot sizes, not by iteration count. Whether the agent is on step five or step forty-five, the prompt fits and the next call has predictable cost and latency.
Quick reference
TL;DR
Carve the window into a small static prefix, a small persistent memory slot, a reserved output slot, and a large working area governed by a hierarchical summary plus soft and hard watermarks that trigger compaction
Think of your suitcase before a 50-day trip. You cannot bring 50 fresh outfits, it will not fit. So you set aside one pocket for documents (the static prefix, never changes), one pocket for essentials (persistent memory), and you keep one corner empty for the souvenir you will pack last (the output). The rest of the suitcase is for clothes, and you cycle them: every five days you bundle the dirty ones into one compact pouch (a level-one summary). When the suitcase looks 80 percent full, you re-roll the pouches. When it looks 95 percent full, you bundle the oldest pouches together into one even smaller bundle. The shape never changes, so you always have room for tomorrow.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short agents that finish in three to five iterations The hierarchical summary and watermark machinery is overhead. A flat append-only context fits and is cheaper to reason about.
Single-tool agents where the tool output is always small The per-tool cap and overflow store add no value when results are bounded by the tool itself. Skip the indirection.
Single-turn assistants with no iteration loop There is no trajectory to budget across. Standard prompt-engineering applies.
Quick scan · 2–4 min
Everything important, quickly.
Lay out the four slots and their sizes, then walk through the hierarchical summary that governs the working area. Cover the per tool result cap and the overflow by reference pattern. Define soft and hard watermarks with the action each one triggers. Close with the property the design enforces: prompt size stays roughly flat across all 50 iterations rather than ramping with step count.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's Claude Opus 4.7 agent loops in production use exactly this slot plus watermark discipline for multi-hour code agents
2LangGraph's MessagesState reducer combined with a summarization node implements the hierarchical summary pattern directly
3Letta (formerly MemGPT) operationalizes the multi-tier working area as named memory blocks with explicit edit tools
4Mem0's per-turn fact injection is the persistent-memory slot in this design, kept small by extraction rather than buffering
5OpenAI's GPT-5.5 Assistants run threads use a similar tail plus summary pattern under the hood for long conversations
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you tune the watermark thresholds for a latency-sensitive vs a cost-sensitive agent?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the failure mode when the working area is too small relative to per-iteration tool output?
Sign in or go Pro to see the approach for this follow-up.
QHow does the slot design change if the model supports prompt caching but only for the prefix?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the 128k window as one undifferentiated pool, then watching the agent crash at iteration 30 when the unbounded scratchpad and uncapped tool results have eaten the whole budget.