How do you keep a long conversation inside the context window without losing state?
A chat keeps growing past the model's context window. Describe strategies to stay under the token limit while preserving the state the model actually needs.
Don't drop the oldest turns blindly. Summarize old history into a rolling summary, keep a verbatim sliding window of recent turns, retrieve relevant old messages on demand, and never truncate the system prompt.
Imagine a long meeting where you can only keep a few pages of notes on your desk at once. If you just throw away the oldest pages, you might toss the page where everyone agreed on the plan. Instead you keep a short summary of the early discussion, the last few pages word for word so the conversation still flows, and a filing cabinet you can dig into when someone references something specific from earlier. And the meeting's ground rules page stays pinned to the desk no matter what — you never throw that away.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every conversational LLM product hits the same wall: the chat keeps growing, and the model can only read a fixed number of tokens. Something has to be left out. The naive instinct — delete the oldest messages until it fits — feels reasonable and is quietly wrong, because the oldest messages are often where the important state lives: the goal the user set, the constraints they gave, the decision everyone agreed on twenty turns back.
The deeper reframe is that the context window is a budget, and managing a conversation is an allocation problem. You're not trying to keep the most recent tokens; you're trying to keep the most valuable tokens. Recency and importance are different axes, and good systems separate them explicitly.
This deep dive builds up the production pattern: a layered context budget with a pinned region, a rolling summary, a verbatim window, and on-demand retrieval. It covers why each layer exists, where summarization quietly degrades, and why even a million-token window doesn't make the problem disappear. The throughline is spending a scarce budget on what the task actually depends on.
Why 'drop the oldest' optimizes the wrong axis
The seductive fix is a queue: when the conversation exceeds the limit, evict from the front until it fits. It's one line of code and it's almost always wrong.
The issue is that importance and recency are different axes. The most recent turn is usually about a detail; the turn where the user said 'I'm allergic to penicillin' or 'the deadline is Friday and we can't move it' might be the oldest one in the buffer. Front-eviction throws those away first while faithfully preserving small talk from five seconds ago.
There's a worse version of this failure. If the system prompt sits at the top of the assembled context — which it usually does — naive front-truncation deletes it first. The model loses the instructions that define its behavior and starts ignoring its own rules, often silently. That single mistake is the subject of an entire class of production bugs.
So the first principle is: truncation is an allocation decision, not a queue operation. You decide which content earns space based on value, not on arrival time. Everything that follows is about making that decision well.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Strategy | What it keeps | What it costs |
|---|---|---|
| Drop oldest | Only recent bytes | Silently loses old decisions and the system prompt |
| Running summary | Gist of old history, cheap | Lossy; can drift if summarized repeatedly |
| Sliding window | Exact recent turns | Nothing older than the window |
| Selective retrieval | Specific relevant old turns | Misses if similarity search doesn't match |
Real products, models, and research that use this idea.
- LangChain's ConversationSummaryBufferMemory keeps recent turns verbatim and rolls older ones into a running summary.
- Claude and ChatGPT apps summarize long conversations into a compact memory so older context survives past the window.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour rolling summary slowly drifts from what actually happened over a very long chat. Why, and how do you fight it?
Discuss compounding loss from summarizing summaries, regenerating from source turns where feasible, pinning hard facts as structured fields, and validating that key state survives each compaction.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blindly dropping the oldest turns to fit the window, which silently deletes decisions, constraints, and the system prompt that the task still depends on.
60 second bullets to scan on the way to the call.
Why 'drop the oldest turns' loses the wrong thing
How running summarization keeps old state at fewer tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.