Order these context elements from first to evict to never evict when the budget tightens
- 1System prompt with persona, policies, and tool definitions
- 2Older level 1 summaries that have a level 2 covering them
- 3Current user turn and the user's literal question
- 4Optional retrieved chunks below the top 3
- 5Stale tool results that have already been consumed
- 6Rolling summary of older turns
- 7Recent verbatim user assistant turns
Evict consumed content first (stale tool results), then dilution (extra chunks), then content already preserved at a higher level (covered summaries), then the rolling summary, then recent turns.
Picture cleaning out a backpack before a trip when it is too heavy. First you throw out the trash you already used, like empty wrappers from snacks you ate. Next you take out the extra bottles of water when you have enough, because they are nice but not needed. Then you take out the old notes you have already copied into a clean summary, because the summary covers them. Then the summary itself, if it is still too heavy. Then some of the recent letters you wrote, only if it gets really desperate. The map and the trip itinerary stay no matter what, because without them you would not even know where you are going. That last pair is the system prompt and the current question.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Overflow is inevitable in any long-running agent. Conversation history grows, retrieval bursts, tool results accumulate, and at some point the assembled prompt exceeds what fits, even on a model with a million-token window. What separates a good production stack from a brittle one is whether overflow is handled as a documented policy or as an improvisation that happens to fire when the assembler runs out of room.
A documented eviction order is the answer. It encodes which content is most expendable, which is sacred, and what the in-between gradient looks like. When the budget tightens, the assembler walks the order top-down and drops content until the prompt fits. When a postmortem asks 'why did the agent forget X?' the answer is grounded in a policy, not in a bug.
The order matters because the cheap and fast move, head truncation, is wrong. Head truncation drops the system prompt first, which decapitates the agent. The right order drops content in inverse proportion to how load-bearing it is, which lands on a specific seven-tier hierarchy that holds up well across agent designs in 2026.
Why content is not all equally evictable
Each block of content in an assembled prompt sits at a different point on the value per token curve.
Consumed tool results carry near-zero information once their substance has been used to answer something. The assistant read the result, wrote an answer that incorporates the relevant content, and the answer is now in the conversation history. The raw tool result is dead weight. Removing it costs almost nothing.
Retrieved chunks below the top 3 are diminishing-return material. The top 3 usually contain whatever signal retrieval is going to deliver; chunks 4 through k are a recall safety margin that helps when the top 3 happen to miss but otherwise just dilutes the context. Removing them costs a small recall hit on edge inputs.
Lower-level summaries covered by higher-level summaries are direct duplicates. A 3-level hierarchical summary always has the lower level covered by the upper level for the older portions of the conversation. Removing the lower level for the covered range costs nothing in information.
The rolling summary block is a lossy compression of recent turns. Removing it costs some detail about the conversation, but the canonical source, the recent verbatim turns, is still in context.
Recent verbatim user-assistant turns are the canonical conversation record. Removing them costs real information: things the user said and expects to be remembered.
The current user turn and the system prompt carry the call itself. Removing them either fails the request or fails the agent.
A correct eviction order walks this curve from cheapest-loss to most expensive loss.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Tier | What lives here | When it gets evicted |
|---|---|---|
| 1 (first out) | Stale consumed tool results | Almost immediately under any pressure |
| 2 | Retrieved chunks below top-3 | When retrieval is over-fetching |
| 3 | Level-1 summaries covered by level-2 | When the hierarchy has rolled forward |
| 4 | Rolling summary block | When the verbatim recent turns are more valuable |
| 5 | Recent verbatim turns (oldest first) | Only when truly desperate |
| 6 (sacred) | Current user turn | Never; reject the request instead |
| 7 (sacred) | System prompt and tool defs | Never; the agent is not the agent without it |
Real products, models, and research that use this idea.
- LangGraph's MessagesPlaceholder + trim_messages utility lets you declare eviction order on the message list; widely used in 2026 agent stacks.
- Claude Code's compaction step evicts consumed tool results and older summary levels first while keeping the active task block and CLAUDE.md sacred.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does citation-aware eviction differ from this static order?
Citation-aware eviction tags retrieved chunks that the assistant has cited as elevated in priority, moving them up tiers. Static order does not distinguish; citation-aware does. Tradeoff is implementation complexity vs accuracy of the eviction decision.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Evicting recent verbatim turns before the rolling summary. The summary is the duplicate; the recent turns are the original. Evict the lossy abstraction first when forced.
60 second bullets to scan on the way to the call.
The two tiers that are never evicted
Why stale consumed tool results are tier 1
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.