1System prompt with persona, policies, and tool definitions
2Older level-1 summaries that have a level-2 covering them
3Current user turn and the user's literal question
4Optional retrieved chunks below the top 3
5Stale tool results that have already been consumed
6Rolling summary of older turns
7Recent verbatim user-assistant turns
When the assembled prompt is about to overflow the model's context window, the budget has to give somewhere. A documented eviction order tells the system what to drop first.
The principle is to evict in inverse order of load-bearingness. Anything the model has already consumed, anything that is duplicated by a higher-level abstraction, anything optional, those go first. Things that are sacred to the current call go last or never.
The canonical order from first to evict to never-evict:
1. **Stale tool results that have already been consumed.** A previous turn used the result to answer a question and the answer is now in the conversation history. The raw result is dead weight.
2. **Optional retrieved chunks below the top 3.** Top-3 is usually where the signal lives; the rest is dilution.
3. **Older level-1 summaries that have a level-2 covering them.** The level-2 already contains what level-1 said; keeping both is duplicate.
4. **Rolling summary of older turns.** The summary is a compressed view; if budget is desperate, drop it before dropping the originals.
5. **Recent verbatim user-assistant turns.** Recency matters; only evict when truly out of options.
6. **Current user turn.** The question itself. Never evict.
7. **System prompt.** Persona, safety, tool definitions. Never evict.
The order above is a documented policy, not a heuristic that fires per call. Documented means the team can answer 'why did the agent forget X?' with 'because the budget tightened and X was below this line in the eviction policy.'
**Eviction order is a documented policy, not an improvised compromise.** The order encodes which content is most expendable and which is sacred. From first to evict to never-evict:
### 1. Stale tool results that have been consumed
A tool result is consumed when its substantive content has been written into the conversation (the assistant has answered a question using it and the answer is now in the turn history). The raw result is now redundant, the answer carries the relevant information forward. Stale tool outputs are pure overhead. Drop them first.
### 2. Optional retrieved chunks below the top 3
For RAG calls, the top 3 chunks (after rerank) usually contain the signal. Chunks 4 through k are dilution: they were retrieved because the recall safety margin asked for them, not because they were needed. When budget tightens, drop chunks k-3 down to 3. This also helps with lost-in-the-middle effects since the kept chunks are stronger signal.
### 3. Older level-1 summaries that have a level-2 covering them
In a hierarchical summary scheme, level-1 (recent abstract) is rolled up into level-2 (older condensed) as conversation grows. Once a stretch of conversation is covered at level-2, the level-1 summary for that stretch is duplicate. Drop it.
### 4. Rolling summary of older turns
If the budget is still tight after the above, drop the rolling summary itself. The summary is a lossy compression of recent history; keeping the verbatim recent turns is more valuable than keeping a lossy abstraction of older turns.
### 5. Recent verbatim user-assistant turns
Only evict here when truly out of options. Recent turns carry context the user expects the agent to remember; evicting them produces visible 'goldfish' behavior. Trim oldest-first within the recent block.
### 6. Current user turn
The question itself. Never evict. If you cannot fit the user's literal question, the request must be rejected with a clear error, not silently truncated.
### 7.
Quick reference
TL;DR
Evict consumed content first (stale tool results), then dilution (extra chunks), then content already preserved at a higher level (covered summaries), then the rolling summary, then recent turns.
Picture cleaning out a backpack before a trip when it is too heavy. First you throw out the trash you already used, like empty wrappers from snacks you ate. Next you take out the extra bottles of water when you have enough, because they are nice but not needed. Then you take out the old notes you have already copied into a clean summary, because the summary covers them. Then the summary itself, if it is still too heavy. Then some of the recent letters you wrote, only if it gets really desperate. The map and the trip itinerary stay no matter what, because without them you would not even know where you are going. That last pair is the system prompt and the current question.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-turn calls with no history and a small retrieval block Overflow is unlikely and the order is unnecessary overhead; a simple budget check is enough
Strict-context APIs where the user controls every token The user has already decided what goes in; you should not silently evict on their behalf
Stateless one-shot agents with no conversation memory Tiers 3-5 do not exist in that shape; a simpler 'sacred system + sacred input vs evictable retrieval' is enough
Quick scan · 2–4 min
Everything important, quickly.
4 minutes: state the seven tiers in order, name the two sacred ones, explain why summary evicts before recent verbatim turns, and call out output token reservation as the precondition.
Side-by-side
Tier
What lives here
When it gets evicted
1 (first out)
Stale consumed tool results
Almost immediately under any pressure
2
Retrieved chunks below top-3
When retrieval is over-fetching
3
Level-1 summaries covered by level-2
When the hierarchy has rolled forward
4
Rolling summary block
When the verbatim recent turns are more valuable
5
Recent verbatim turns (oldest first)
Only when truly desperate
6 (sacred)
Current user turn
Never; reject the request instead
7 (sacred)
System prompt and tool defs
Never; the agent is not the agent without it
Real-world examples
Real products, models, and research that use this idea.
1LangGraph's MessagesPlaceholder + trim_messages utility lets you declare eviction order on the message list; widely used in 2026 agent stacks.
2Claude Code's compaction step evicts consumed tool results and older summary levels first while keeping the active task block and CLAUDE.md sacred.
3Cursor's background agents apply a similar tiered eviction when long sessions approach Claude Opus 4.7's context limit.
4LlamaIndex's ContextChatEngine has explicit token_limit and trim policies that follow the same hierarchy (system → recent → summary → retrieved).
5MemGPT and Letta (the 2026 rename) implement a paged-memory model with explicit working-context vs archival-storage tiers, which is the eviction order encoded as storage layers.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does citation-aware eviction differ from this static order?
Sign in or go Pro to see the approach for this follow-up.
QWhen you compact via summarization mid-session (like Claude Code or Cursor), how does that interact with the eviction order?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes about this order if the agent has persistent semantic memory that lives outside the context window?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Evicting recent verbatim turns before the rolling summary. The summary is the duplicate; the recent turns are the original. Evict the lossy abstraction first when forced.