Order these context elements from first to evict to never evict when the budget tightens
- 1System prompt with persona, policies, and tool definitions
- 2Older level 1 summaries that have a level 2 covering them
- 3Current user turn and the user's literal question
- 4Optional retrieved chunks below the top 3
- 5Stale tool results that have already been consumed
- 6Rolling summary of older turns
- 7Recent verbatim user assistant turns
Evict consumed content first (stale tool results), then dilution (extra chunks), then content already preserved at a higher level (covered summaries), then the rolling summary, then recent turns.
Picture cleaning out a backpack before a trip when it is too heavy. First you throw out the trash you already used, like empty wrappers from snacks you ate. Next you take out the extra bottles of water when you have enough, because they are nice but not needed. Then you take out the old notes you have already copied into a clean summary, because the summary covers them. Then you take out the summary itself, if it is still too heavy. Then you take out some recent letters you wrote, but only if you are desperate. The map and the trip itinerary stay no matter what, because without them you would not even know where you are going. That last pair is the system prompt and the current question.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Overflow is inevitable in any long-running agent. Conversation history grows, retrieval bursts, and tool results accumulate. At some point the assembled prompt exceeds what fits, even on a model with a million-token window. A good production stack handles overflow with a documented policy. A brittle one improvises when the assembler runs out of room.
A documented eviction order is the answer. It encodes which content is most expendable, which is sacred, and what the in-between gradient looks like. When the budget tightens, the assembler walks the order top-down and drops content until the prompt fits. When a postmortem asks 'why did the agent forget X?' the answer is grounded in a policy, not in a bug.
The order matters because the cheap and fast move, head truncation, is wrong. Head truncation drops the system prompt first, which decapitates the agent. The right order drops content in inverse proportion to how important it is. That gives a specific seven-tier hierarchy that holds up well across agent designs.
Why content is not all equally evictable
Each block of content in an assembled prompt sits at a different point on the value per token curve.
Consumed tool results carry near-zero information once their substance has been used to answer something. The assistant read the result, wrote an answer that incorporates the relevant content, and the answer is now in the conversation history. The raw tool result is dead weight. Removing it costs almost nothing.
Retrieved chunks below the top 3 are diminishing-return material. The top 3 usually contain whatever signal retrieval is going to deliver. Chunks 4 through k are a recall safety margin. It helps when the top 3 happen to miss, but otherwise it dilutes the context. Removing them costs a small recall hit on edge inputs.
Lower-level summaries covered by higher-level summaries are direct duplicates. A 3-level hierarchical summary always has the lower level covered by the upper level for the older portions of the conversation. Removing the lower level for the covered range costs nothing in information.
The rolling summary block is a compression of older turns that are no longer in context verbatim. Removing it loses that older history, which is a real cost. It still ranks below the recent turns because the next answer depends mostly on the live exchange. Older history is less likely to matter to it.
Recent verbatim user-assistant turns are the canonical conversation record. Removing them costs real information: things the user said and expects to be remembered.
The current user turn and the system prompt carry the call itself. Removing them either fails the request or fails the agent.
A correct eviction order walks this curve from cheapest loss to most expensive loss.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Tier | What lives here | When it gets evicted |
|---|---|---|
| 1 (first out) | Stale consumed tool results | Almost immediately under any pressure |
| 2 | Retrieved chunks below top-3 | When retrieval is over-fetching |
| 3 | Level-1 summaries covered by level-2 | When the hierarchy has rolled forward |
| 4 | Rolling summary block | When older history matters less than the recent turns |
| 5 | Recent verbatim turns (oldest first) | Only when truly desperate |
| 6 (sacred) | Current user turn | Never; reject the request instead |
| 7 (sacred) | System prompt and tool defs | Never; the agent is not the agent without it |
Real products, models, and research that use this idea.
- LangChain core's trim_messages trims a message list by message count or token budget and can keep the system message and the latest turns; a full tiered order is built on top of it in the assembler.
- Anthropic's context editing (clear_tool_uses) clears old tool results once input passes a trigger (100,000 tokens by default) while keeping the 3 most recent tool uses, which is tier 1 of this order applied server-side.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does citation-aware eviction differ from this static order?
Citation-aware eviction tags retrieved chunks that the assistant has cited as elevated in priority, moving them up tiers. Static order does not distinguish; citation-aware does. The trade-off is implementation complexity vs accuracy of the eviction decision.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Evicting recent verbatim turns before the rolling summary. The recent turns are what the next answer depends on most; the summary holds older, less relevant history. Drop the older history first when forced.
60 second bullets to scan on the way to the call.
The two tiers that are never evicted
Why stale consumed tool results are tier 1
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.