Why cap tool output size at the tool boundary instead of letting the model deal with it?

Flashcard·Medium·4.0 · 0·~30s·Asked atBaseten Inflection Ai Jane Street

Attempt it

Returning a 50K-token web page into context burns budget for content the model will mostly ignore, and silently triggers lost-in-the-middle across the trajectory. The model cannot 'choose what to read' inside a tool result; it sees the entire blob. Capping at the tool boundary gives the application control: truncate to the top-N most relevant sections, prepend metadata (URL, title, retrieved-at), or run a small summarizer pass before handing back. The agent can always re-call the tool with a more specific query if needed. Tool wrappers that respect a max output tokens budget are one of the single biggest reliability wins for agent systems.

Your answer

Tools in an agent system can return enormous outputs. A web-search tool that fetches a page can produce tens of thousands of tokens. A file-read tool over a log file can produce hundreds of thousands. An API call to a database can return a giant JSON document. If the tool wrapper returns this blob unchanged into the model's context, two bad things happen. The token budget is consumed by content the model will mostly ignore, and the relevant tokens end up buried in the middle of a long context where the **lost-in-the-middle** effect kicks in. The model cannot "scroll" through a tool result and pick out the useful section; it sees the entire blob as one chunk in the conversation. The canonical fix is to cap tool output at the tool boundary, inside the wrapper code that calls the underlying service. The wrapper can truncate to the top-N most relevant sections, prepend useful metadata (URL, title, the time the result was fetched), or run a small summarization pass before handing back to the model. If the agent decides it needs more, it can call the tool again with a narrower query or with an explicit "give me page 2" argument. This shifts the decision of *what to read* from the model, which has poor information about the source format, to the wrapper, which knows exactly how the underlying response is structured.

Tool wrappers in agent systems sit between the model and the outside world: a search API, a file-read function, a database query, a shell command. The wrapper is the right place to enforce a token budget on tool output, and the reason is structural rather than stylistic. The model cannot skim a tool result. When a 50K-token web page lands in context as one tool_result message, the model sees the whole thing at once and has to allocate attention across all of it. The relevant content is buried in the middle of the trajectory, and the **lost-in-the-middle** curve guarantees attention skips it. Meanwhile, the entire blob is billed at input-token rate and erodes the remaining context budget for the rest of the trajectory. Capping at the wrapper turns this from a model problem into an application problem, which is the right layer to solve it. The wrapper knows the source format: it can parse the HTML, strip nav and footer, and return only the article body. It can truncate to the top-N sections by a relevance heuristic. It can prepend structured metadata, URL, title, retrieved-at, so the model knows what it got. For very large sources, it can run a small summarizer LLM and return the summary plus a handle (a URL fragment, a chunk id) that the agent can use to fetch more. The agent loses nothing important. If it decides the truncated result is insufficient, it can re-call the tool with a more specific query, request a different page, or ask for a different section. The cost of one extra tool call is far less than the cost of one polluted context window. Production agent systems treat tool-output capping as load-bearing. Anthropic's Claude Code, Cursor, Devin, and every coding agent in 2026 enforces strict per-tool token budgets. Without that discipline, multi-step agent trajectories collapse under their own context weight by step five.

TL;DR

Capping tool output at the wrapper keeps a single tool call from torching the context budget and triggering lost-in-the-middle, and the wrapper has structured-format knowledge the model lacks.

Memory aid

Easy to grasp

Picture a research assistant who can fetch books from a library for you, and your desk only fits five books at a time. If the assistant dumps every book they can find onto the desk, half of them slide off and the ones underneath are buried. A good assistant skims each book first, hands you the relevant chapter, and writes the rest of the title on a sticky note so you can ask for more if you need it. The tool wrapper is that assistant. The model is you at the desk. Without a cap at the wrapper, one tool call can bury everything else on the desk, including the question you were trying to answer. With a cap, the desk stays usable and the agent can ask for more on demand.

Key concepts

Context Engineering Agents

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Tool wrappers are the seams between the model and the outside world. They take the model's structured tool call, invoke an external service, and hand the result back into context. The single most consequential design decision at this seam is the policy for output size, how much of the underlying response actually reaches the model, and in what shape.

Getting this wrong is one of the most common reasons agent systems break under their own weight. A few uncapped tool calls into a long trajectory and the context is full of pages of HTML the model is not really reading, the user's original question has slid into the lost-in-the-middle trough, and the cost per turn has quadrupled. This deep dive walks through why capping at the tool boundary is the canonical pattern, how to choose the right capping strategy per tool, and what failure modes show up when the cap is too tight or too loose.

Why the model cannot self-truncate a tool result

It is tempting to imagine the model handling tool-result size on its own. Just give it the whole response and trust it to focus on the useful parts. This intuition is wrong, and the reason is mechanical.

The model has no operation for "read only the first N tokens of the tool result." The result is a single tool_result message in the conversation, and from the model's perspective it is one chunk of text alongside the assistant's turns and the user's turns. Attention spreads across the whole message proportional to learned weights, not according to a relevance filter the model can apply on the fly. Whatever the wrapper returns is what gets attended to.

This is fundamentally different from how a human runs a tool. A human running cat large_file.txt sees the terminal start scrolling, hits Ctrl-C, and runs head or grep instead. The model has no such interrupt. It does not see the result streaming in, it sees a completed message, all at once, after the tool call finishes. The decision of how much to read has already been made by the time the model gets the result.

This is why the wrapper, not the model, is the right place to enforce size limits. The wrapper has the choice of what to return. The model only has the choice of how to interpret what was returned.

The lost-in-the-middle interaction

Three capping strategies and when each fits

Signaling truncation and enabling continuation

Per-tool caps and the production discipline

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's Claude Code caps Bash and file-read tool outputs at a few thousand tokens by default and appends a 'use grep for more' hint when truncated.
Cursor's agent mode wraps file-read with a chunked by symbol reader that returns one function or class at a time rather than the whole file.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you decide between hard truncation, top-N extraction, and summarization for a given tool?

It depends on the source structure and the cost budget. Structured sources (JSON APIs, databases) get field projection, return only the keys the agent asked about. Semi-structured sources (HTML pages, markdown docs) get top-N extraction by section relevance, often with a small reranker. Truly unstructured very-large sources (log files, transcripts) get a summarizer pass when latency allows, or a chunked-pagination tool when it does not. Hard byte truncation is the last resort and should always be paired with a 'truncated' marker.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating tool outputs as raw passthroughs and assuming the model will handle truncation on its own, when in fact the model sees the entire blob and burns budget over it.

60 second bullets to scan on the way to the call.

State why the model cannot self-truncate a tool result
Connect tool-output bloat to the lost-in-the-middle effect

Primary sources. Browse if you want the original framing.