Pick the right fallback when the assembled prompt overflows the model's context window

MCQ·Medium·4.0 · 0·~1 min·Asked atDatabricks OpenAI Anthropic

Attempt it

When the assembled prompt is bigger than the model's context window, something has to go. The question is which of four common approaches is the right one. **Option 0, reject the request.** Sometimes correct (if the user pasted a 100k-token document), but usually wrong. Most overflow is caused by retrieval volume, summary growth, or accumulated tool results, things the user did not control. Punishing the user for the system's assembly is user-hostile. **Option 1, silently truncate from the start.** Catastrophic. The start of the prompt is the system prompt, persona, safety rules, and tool definitions. Head truncation decapitates the agent. Symptoms: agents start ignoring brand voice or making unsafe responses under load. Never the right move. **Option 2, apply a documented eviction order.** Correct. The order is declared in advance: sacred tiers (system prompt, current user question, output reservation) are protected; evictable tiers (extra retrieved chunks, lower-level summaries, the rolling summary, then recent verbatim turns as a last resort) are trimmed in declared order. The system fails gracefully and the postmortem is debuggable. **Option 3, silently swap to a larger-window model.** Tempting but wrong. The caller did not authorize a model swap. Price changes (a 1M-window model may be 3-5x more expensive). Behavior changes (different model snapshots produce different outputs). The audit trail breaks. Model selection must be an explicit policy, not an emergency fallback. The correct answer is option 2 because it is the only one that is predictable, debuggable, and respects both the user's input and the call's audit trail.

**The correct fallback is a documented eviction order.** The other three options each fail in a specific, predictable way. ### Why reject (option 0) is usually wrong Overflow is mostly the system's responsibility, not the user's. Retrieval over-fetched. Summary grew. Tool results accumulated. The user's literal input is rarely the overflow cause; even when it is, the system can apply eviction to the surrounding context and still serve the request. Hard rejection is the right move only when the user's own input alone exceeds the budget after everything else has been evicted. That is a real failure mode and a clear error is the right response, but it should be the last resort, not the first. ### Why silent head-truncation (option 1) is catastrophic The start of the prompt holds the system prompt: persona, safety policies, tool definitions, prompt cache friendly stable content. Truncating from the start kills the agent's identity and safety guarantees. Production failure mode: agents start violating brand voice or safety policy under high-context load, and the failures only show up at the budget edge. Most naive implementations default to head-truncation because string slicing from the start is the cheapest possible operation. The cost shows up later, when the agent starts misbehaving and the team has to figure out why. ### Why silent model-swap (option 3) is wrong A larger-context model is a different model. It has different pricing (often 3-5x), different behavior characteristics (some are slower, some are weaker on reasoning, some have different safety calibration), and different rate limits. Swapping silently means the caller pays more, gets different behavior, and cannot audit what happened. Model selection is a policy decision; a per-call automatic fallback is operating outside that policy. The right pattern is to declare a model ladder explicitly.

Quick reference

TL;DR

The right overflow fallback is a documented eviction order: sacred (system prompt + current turn + output reservation) protected, evictable (extra chunks, summary levels) trimmed in declared order.

Memory aid

Easy to grasp

Imagine a suitcase that will not close. Three bad options. First, refuse to go on the trip until the traveler removes things; rude, and they did not pack most of it themselves. Second, just sit on the lid and squish whatever happens to be at the bottom; you smash the medicine and the passport. Third, secretly trade the suitcase for a bigger one without telling them; they did not budget for a bigger suitcase. The right way is a packing list with priorities. The passport and medicine always stay. The third pair of shoes, the extra snacks, those come out first, in that order, until the suitcase closes. That order is the eviction order, and it is decided before the trip, not on the curb.

Key concepts

Context engineering LLM frameworks

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 minutes: state the right answer is documented eviction, walk one failure mode for each of the three wrong answers, and name sacred tiers (system prompt + current user turn + output reservation).

Side-by-side

Option	Failure mode	When (if ever) right
Reject the request	Blames the user for system-side overflow	Only when user input alone exceeds budget after max eviction
Silent head-truncate	Drops system prompt and safety rails	Never; antipattern
Documented eviction order	Quality degrades gradually under budget pressure	Default for any production system
Silent model swap	Changes price and behavior without notice	Never silent; ok with an explicit declared ladder and logging

Real-world examples

Real products, models, and research that use this idea.

Anthropic Claude Code applies a compaction step before overflow: stale tool results and older summary levels are evicted from working context while CLAUDE.md and the active task block stay sacred.
LangGraph's trim_messages with strategy='last' preserves recent turns and the system prompt while trimming older history.
LlamaIndex ContextChatEngine has a token_limit and a documented trim order (system → recent → summary → retrieved) that mirrors the production-correct pattern.
Cursor's background agents declare an explicit model ladder (Claude Sonnet 4.6 → Opus 4.7 → Opus 4.7 with extended context) with logged escalations.
Vercel AI SDK's experimental_telemetry surfaces eviction events so the consumer of the API can audit when a prompt was trimmed and how many tokens were removed.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat does a good error message look like when reject is the correct response?

QHow do you eval an eviction policy in CI?

QWhy does silent model swap on overflow create a non-stationary eval surface?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Picking head-truncation as the simple default. It drops the system prompt first, which is exactly the tier that should be untouchable.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.