Spot the context-engineering error in this tool-result handling code

Spot the error·Medium·4.0 · 0·~2 min·Asked atAnthropic OpenAI Cursor

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The snippet does two things that look reasonable on the surface and that interact catastrophically when combined. First, it appends every tool result to a `results_history` list inside the system prompt. The intuition is that the system prompt sits at the top of the context, in the privileged primacy slot, so tool outputs placed there should get strong attention. The intuition is wrong, because the system prompt grows monotonically with every tool call, and the rest of the conversation gets pushed deeper into the context, into the **lost-in-the-middle** trough. Second, it sends the same system prompt on every turn for prompt-caching benefits. The intuition here is also reasonable, caching is a real cost win, but it cannot coexist with a system block that mutates every turn. Prompt caches are keyed on byte-exact prefixes. The moment a new tool result is appended, the cache key shifts and the cache miss eats the cost. The combined effect is that the code pays full cost on every call (cache always misses) and also gets degraded quality (tool results in the primacy slot push real conversation into the trough). The correct pattern is opposite on both axes. Tool results live in the conversation message sequence, threaded between the assistant's tool call and its next reasoning turn. Stale results get evicted or summarized once they have been consumed by a downstream step. The system prompt stays byte-stable across turns and the cache actually hits.

Two compound errors. First, the snippet mutates the system prompt on every turn by appending tool results. This defeats prompt caching. Anthropic, OpenAI, and Google prompt caches are all keyed on the literal bytes of the prefix. A system block that grows with every tool call has a different byte sequence on every request, so the cache key never matches and the cache rate falls to zero. The line about "prompt-caching benefits" is exactly the part that does not work, the design promises caching while structurally making it impossible. Second, even setting caching aside, the system prompt is the wrong home for tool results. Tool results have a natural home in the conversation message sequence: the assistant's tool_use turn, then the tool_result turn, then the assistant's next reasoning turn. This is how Anthropic's tool-use format, OpenAI's tool_calls, and Google's function-calling format all encode it. Threading the result inline with the reasoning that consumes it places it at the recency end of the context, where attention is reliable. Putting tool results in the system block does the opposite. The system block is at the absolute start of context, which initially feels like a privileged position, but as more tool results pile up, the *non tool result* parts of the conversation (the user question, the assistant's plan, the earlier reasoning) slide deeper into the middle of an ever-longer context. The lost-in-the-middle effect, measured across all modern long-context models, says those middle tokens get attention-starved. A third issue worth naming is the lack of eviction. Tool results have a useful lifetime measured in steps, not in sessions. Once a step has consumed a tool result and produced its output, the raw result is dead weight. The correct pattern is summarize then evict: replace the raw result with a compact pointer ("web_search(query=X) returned 8 results; top three relevant to current step are ...") and free the budget for new observations. The

Quick reference

TL;DR

The snippet shoves growing tool results into the system prompt, which destroys prompt caching and dumps tool outputs into the lost-in-the-middle trough at the same time.

Memory aid

Easy to grasp

Picture a noticeboard at a coffee shop with a sign that says 'this stays the same all day.' If the barista starts pinning every customer's receipt to that board as the day goes on, two things break. The 'stays the same' promise is gone, so any system that was relying on it stops working. And the board fills up so fast that the original menu, the thing customers actually need, is buried under three months of receipts. The system prompt is that noticeboard. Tool results are the receipts. They belong in a different place, and old ones should come down when they are no longer useful.

Key concepts

Context Engineering

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

State both errors and their interaction. Error one: appending to the system prompt destroys the cache key, so the cache-rate claim is structurally impossible. Error two: tool results in the system block push real conversation into the lost-in-the-middle trough. Add the third issue, missing eviction, and explain the summarize then evict pattern. Close with the right structural design: byte-stable system prompt, tool results threaded in the conversation tail, eviction at the wrapper or reducer layer.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's Claude tool-use format threads tool_result blocks directly into the conversation, with the system prompt reserved for stable persona and tool schemas.
OpenAI's Responses API places tool_calls and their outputs as separate message events in the response stream, never as system-prompt content.
LangGraph stores tool observations in typed state with explicit reducer functions for eviction, separating durable state from prompt text.
Letta (formerly MemGPT) exposes memory-management tools the model can use to move observations between in-context and out of context tiers, rather than letting the prompt grow.
Cursor and Claude Code summarize older tool results into compact references once subsequent steps no longer need the raw output, freeing budget for new observations.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you wanted some tool results to be durable across many turns, what is the right way to do it without breaking the system-prompt cache?

QWhat goes wrong if you fix the caching issue by adding a cache breakpoint after the mutating section?

QHow do production agent frameworks signal to the model that a tool result has been summarized or evicted?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Conflating 'I want this content to be cached' with 'put it in the system prompt', caching only works while the block is byte-stable, which a growing tool log never is.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.