How does a sliding window memory keep a chat under budget, and what does it always lose?

Flashcard·Easy·4.0 · 0·~30s·Asked atAlibaba Infosys Locus

Attempt it

Sliding-window memory keeps only the last N turns (or last K tokens) of the conversation and drops everything older verbatim. It is cheap, predictable, and adds zero summarization cost per turn. The trade-off is hard: anything older than the window is gone. Long-range facts a user mentioned earlier (their name, a constraint set at the start, a decision made ten turns ago) silently fall off and the model behaves as if the user never said them.

Your answer

**Sliding-window memory** is the simplest possible way to keep a chatbot's conversation history under a token budget. It keeps the last N turns (or the last K tokens) of the conversation and drops everything older. There is no compression, no summary, no replacement; older turns just stop being included in the prompt. The mechanism is a FIFO queue over turns or tokens. When a new turn comes in, it goes to the front; if adding it would push the total over the budget, the oldest turn at the back falls off. The model only ever sees what is currently in the queue. The wins are operational. It adds zero LLM-call cost per turn (no summarizer is run). It is deterministic: the same conversation history produces the same model input. It is dead-simple to implement (a list slice) and easy to debug (the model input is exactly the last N turns). The one thing it cannot retain, by construction, is **anything older than the window**. A user who mentioned their name, set a constraint, or made a decision ten turns ago loses that information the moment the window slides past. The model then behaves as if that turn never existed. There is no graceful decay or partial preservation. The information is either present verbatim or completely absent.

**Sliding-window memory** keeps only the last N turns (or last K tokens) of the conversation verbatim and drops everything older. There is no summary, no extracted-fact store, no replacement of any kind. It is a FIFO over the conversation: turns enter at the front, fall off at the back when the budget is full. The pattern is the simplest possible answer to bounded-context. The wins are operational and unambiguous. **Zero LLM-call cost** per turn beyond the main generation, because no summarizer is invoked. **Fully deterministic** behavior: the same conversation history produces the same model input on every call. **Trivial implementation**, often a one-line list slice. **Easy to debug**, since the model input is exactly the last N turns and nothing else. The one thing it cannot retain, by construction, is **anything older than the window**. A user who told the bot at turn three "I am vegetarian," mentioned their company at turn five, or set a budget at turn seven loses each of those facts the moment the relevant turn falls off the back of the queue. The model does not approximate the missing fact, does not infer it from later context, and does not have any way to recover it. It simply behaves as if the turn never happened. This is the **hard-cutoff failure mode**. Unlike rolling summarization, which degrades old content gracefully, sliding window is binary: present verbatim or completely absent. For applications where old facts never recur (single-task tools, transactional flows), the failure is benign. For chat assistants and agents where user-stated preferences might come up at any time, it is catastrophic. The choice of N (or K tokens) trades reach against cost. Larger N keeps more context but eats budget. Production typical values land between 5 and 20 turns, depending on how far back pronouns and definite references typically point in the workload.

TL;DR

Sliding-window memory keeps only the last N turns verbatim and drops older ones. It is cheap and deterministic, but anything older than the window vanishes with no summary or substitute to replace it.

Memory aid

Easy to grasp

Imagine a fridge with a strict rule: only the five most recent items stay, anything older gets thrown out. No leftover gets summarized into a note on the door, no fridge inventory gets kept somewhere else. It is simple, the fridge never overflows, but the milk you bought last week is gone the moment the sixth item arrives. A sliding-window chat memory works the same way. The last few turns are kept word for word, anything older drops off the back, and nothing replaces them. The bot behaves as if those old turns never happened.

Key concepts

Context Engineering Agents

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Sliding-window memory is the simplest answer to the question "how do I keep a chatbot's conversation history under a token budget?" Keep the last N turns, drop the rest. The mechanism is so straightforward that it feels barely worth a name, but understanding what it does and does not do is the foundation for understanding every more sophisticated memory pattern.

This deep dive walks through the mechanism, the wins, the single hard limit, and how the pattern fits into the broader memory composition that most production stacks use.

The mechanism: FIFO over turns or tokens

A sliding window is a FIFO queue. Each new turn (user or assistant) is appended at the front. When the total exceeds the budget, the oldest turn at the back is removed. The model sees exactly whatever is currently in the queue at the time of the call.

The budget can be expressed in two units. Turn count: keep the last N turns regardless of size. Token count: keep the last K tokens regardless of how many turns that is. Production stacks typically use token count because turn sizes are highly variable (a user who pastes a 5,000-token document followed by short questions will blow a turn-count budget while a token-count budget evicts naturally).

There is no other moving part. No summarizer runs. No extracted facts are stored. No analysis is done on what is being dropped. The eviction happens silently, deterministically, and cheaply.

Implementation is trivially small. In Python it is essentially a list slice; in any other language it is the same. The complexity all sits in deciding the value of N or K, which depends on the workload's typical reference distance.

What the pattern gets right

The single thing it cannot retain

Choosing N and composing with other layers

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LangChain's ConversationBufferWindowMemory is the canonical sliding-window implementation.
LlamaIndex's ChatSummaryMemoryBuffer falls back to a sliding window when summarization is disabled.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you pick the window size N for a chat application?

Sample real conversations and measure the typical reference distance: how many turns back does a pronoun or definite reference usually point? Set N to comfortably exceed the 90th percentile of that distance. For most chat workloads, N lands between 5 and 20 turns; for code-heavy workloads with long pasted blocks, K tokens is a better unit than N turns.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Thinking the dropped turns get summarized somewhere. In a pure sliding window, no summary is created and the dropped content is genuinely lost.

60 second bullets to scan on the way to the call.

Define sliding-window memory in one sentence
Explain the FIFO mechanism and what triggers an eviction

Primary sources. Browse if you want the original framing.