Spot the context engineering error in this tool result handling code

Spot the error·Medium·4.0 · 0·~2 min·Asked atSnowflake Typeface

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The snippet does two things that look reasonable on the surface but interact badly when combined. First, it appends every tool result to a `results_history` list inside the system prompt. The intuition is that the system prompt sits at the top of the context, in the privileged primacy slot (the strong-attention start). So tool outputs placed there get strong attention. The intuition is wrong. The system prompt grows with every tool call, and the rest of the conversation gets pushed deeper into the context, into the **lost-in-the-middle** trough. Second, it sends the same system prompt on every turn for prompt-caching benefits. But prompt caches are keyed on byte-exact prefixes, and this system block changes every turn. The moment a new tool result is appended, the prefix changes at the list, and the list plus everything after it misses the cache. The combined effect is higher cost on every call and lower answer quality. The correct pattern is opposite on both axes. Tool results live in the conversation message sequence, between the assistant's tool call and its next reasoning turn. Stale results get evicted or summarized once consumed. The system prompt stays byte-stable across turns and the cache actually hits.

There are two compound errors. First, the snippet mutates the system prompt on every turn by appending tool results. This defeats prompt caching. Anthropic, OpenAI, and Google prompt caches are all keyed on the literal bytes of the prefix. A system block that grows with every tool call changes at the list on every request. So the list and everything after it miss the cache. Only the static text before it can still hit. The claimed "prompt-caching benefits" do not materialize. Second, even setting caching aside, the system prompt is the wrong home for tool results. Tool results belong in the conversation message sequence: the assistant's tool_use turn, then the tool_result turn, then the assistant's next reasoning turn. Anthropic's, OpenAI's, and Google's tool-calling formats all encode it this way. That places each result at the recency end of the context, where the model uses information reliably. In the system block, a growing list does the opposite. As tool results pile up, the user question, the assistant's plan, and the earlier reasoning slide deeper into the middle of an ever-longer context. There, the lost-in-the-middle effect makes them harder to use. A third issue is the lack of eviction. Once a step has consumed a tool result, the raw result is dead weight. The correct pattern is summarize then evict: replace the raw result with a compact pointer ("web_search(query=X) returned 8 results; top three relevant to current step are ...") and free the budget for new observations.

TL;DR

The snippet puts growing tool results into the system prompt, which breaks prompt caching and pushes the rest of the conversation into the lost-in-the-middle trough at the same time.

Memory aid

Easy to grasp

Picture a noticeboard at a coffee shop with a sign that says 'this stays the same all day.' If the barista starts pinning every customer's receipt to that board as the day goes on, two things break. The 'stays the same' promise is gone, so any system that was relying on it stops working. And the board fills up so fast that the original menu, the thing customers actually need, is buried under a day's worth of receipts. The system prompt is that noticeboard. Tool results are the receipts. They belong in a different place, and old ones should come down when they are no longer useful.

Key concepts

Context Engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

This snippet is short, but it violates three distinct context engineering principles at once. The violations interact in a way that makes the design strictly worse than any one violation alone.

The design says: (1) tool results go in a persistent list in the system prompt, (2) tool results are never removed, and (3) the same system prompt is sent every turn for prompt-caching benefits. Claims 1, 2, and 3 cannot all be true at the same time. The system prompt cannot be both "the same on every turn" and "the place where new tool results are appended on every turn."

Why the prompt-caching claim is structurally impossible

Prompt caching on Anthropic, OpenAI, and Google all share one property: the cache key is the literal byte sequence of the cached prefix. When the prefix matches byte for byte across two requests, the second request reuses the cached state. When it differs by even one character, the cache misses and the model recomputes from scratch.

The snippet says the system prompt is sent the same every turn. The snippet also says tool results are appended to a list inside the system prompt on every turn. Both cannot be true. After the first tool call, the system prompt has one extra result. After the second, it has two. The byte sequence changes at the list on every request, so the list and everything after it miss the cache. Only the static text before the list can still hit.

The error survives because each half of the design looks correct in isolation. "Cache the system prompt for cost savings" is a textbook-correct move. "Put persistent state in the system prompt so it is always available" is plausible-sounding. The combination is incoherent.

The fix is to move the mutating content out of the cached span. Tool results go into the conversation message sequence, in the uncached tail. The system prompt becomes truly byte-stable and the cache actually hits.

Why the system prompt is the wrong home for tool results regardless

The missing eviction policy

The fix and the framework lesson

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's Claude tool-use format threads tool_result blocks directly into the conversation, with the system prompt reserved for stable instructions and tool schemas sent in the separate tools parameter.
OpenAI's Responses API represents tool calls and their outputs as function_call and function_call_output items in the input sequence, never as system-prompt content.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you wanted some tool results to be durable across many turns, what is the right way to do it without breaking the system-prompt cache?

Store the durable observations in agent state (LangGraph typed state, a memory store, a vector index) rather than in the prompt text. Expose a recall_observation(id) tool the model can call when it needs an older result. The system prompt stays cacheable, and the model fetches durable content on demand into the recency end of context where attention is strong.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Conflating 'I want this content to be cached' with 'put it in the system prompt'. Caching only works while the block is byte-stable, which a growing tool log never is.

60 second bullets to scan on the way to the call.

Identify which property of prompt caching the snippet violates
Explain why the system prompt is the wrong location for mutating tool results

Primary sources. Browse if you want the original framing.