Bound each tool result at the tool boundary and stash overflow out of band. The bloat is input tokens from raw tool payloads, not anything the model is choosing to write.
Imagine a detective taking notes during interviews. After ten interviews, the notebook is so thick the detective can barely lift it. Most pages are coffee-shop menus and parking receipts the witnesses happened to be holding. The fix is not buying a bigger notebook. The fix is for the assistant who hands notes over to keep one short paragraph per interview and file the receipts in a drawer with a label. If the detective needs a receipt later, they can ask for it by label. That is what capping tool results does for an agent: the drawer is out of band storage, and the label is the reference token in context.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent loops are deceptively expensive. Each iteration looks like a clean cycle (model plans, tool runs, result returns, model continues). But the context window is the accumulator, and most production loops have no garbage collection. A tool that returns 8KB of incidental data per call quietly grows the prompt by 2,000 tokens per iteration. Those tokens are billed and re-attended on every subsequent call in the trajectory.
The problem is structural rather than tactical. Three of the four options treat the bloat as something to be managed at the model layer: enlarge the window, tighten temperature, ask politely for brevity. None of those touch the mechanism, because the model has no say in what its own inputs look like. The harness composes the prompt. The tool wrapper produces the payload. The fix has to live there.
The correct option, B, addresses what the prompt is made of. It caps each tool result at a known size, stores overflow somewhere addressable, and lets the agent pull more on demand.
The accumulator: what happens per iteration
Take a search tool that returns the full HTML of the top result. The tool runs, the harness appends the result to the message list as a tool result, and the agent makes its next call. The model sees the full HTML on iteration 2. It also sees it on iteration 3. It sees it again on iteration 4. Each iteration carries every prior tool result forward, because the standard tool-use message format (OpenAI, Anthropic, Vercel AI SDK) appends rather than replaces.
Why this hurts more than it looks
The damage is on three axes at once. Cost scales linearly in the cumulative tool result size. Each call pays the input rate on the running total. Latency grows the same way because time to first token is dominated by prompt length, especially for non-cached tokens. Quality degrades through the lost-in-the-middle effect: as the prompt grows, the user's actual task drifts toward the centre where attention is weakest.
A worked example
A ten-iteration trajectory with 8KB of raw tool output per iteration accumulates around 20K tokens of low-signal input by the end. Summed over the 10 calls (2K + 4K + ... + 20K), you have paid input rate on roughly 110K tokens. That handles only 80KB of source material that the agent could have summarised once at 2K tokens.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Tool wrapper that caps and stores overflow out of band
from uuid import uuid4
MAX_TOOL_TOKENS = 1024 # roughly 4KB of text
def wrap_tool_result(raw: str, store) -> dict:
if len(raw) <= MAX_TOOL_TOKENS * 4: # ~4 chars/token
return {"kind": "inline", "content": raw}
head = raw[: MAX_TOOL_TOKENS * 2]
tail = raw[-MAX_TOOL_TOKENS * 2 :]
ref_id = uuid4().hex[:8]
store.put(ref_id, raw)
return {
"kind": "capped",
"head": head,
"tail": tail,
"elided_bytes": len(raw) - len(head) - len(tail),
"ref_id": ref_id,
"fetch_hint": f"call fetch_blob(ref_id='{ref_id}', offset=..., length=...) to read more",
}Real products, models, and research that use this idea.
- Anthropic's context editing beta (clear_tool_uses_20250919) clears older tool results on the server once input passes a token trigger, and keeps the most recent ones.
- Anthropic's post "Effective context engineering for AI agents" (September 2025) recommends just-in-time retrieval through lightweight references such as file paths and links, instead of loading full data up front.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is a sensible eviction policy for the ids stashed out of band when the agent runs for hours?
Talk about reference counting versus time-based decay, and how summarisation can free ids whose content has been distilled into a memory block.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Asking the model to be brief controls only output tokens. The bloat lives on the input side, where the model has no agency.
60 second bullets to scan on the way to the call.
What is the difference between input-side bloat and output-side bloat in an agent loop?
Where does a per-tool size cap belong: tool wrapper, system prompt, or model behaviour?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.