Distinguish short term vs long term agent memory and name a failure mode specific to each
Explain the difference between short term and long term memory in an agent system. Give one failure mode that is unique to each type.
Short-term memory is the live context window for the current task; long-term memory is an external store retrieved as needed. Each fails differently: overflow versus staleness.
Imagine you are cooking from a recipe. Short-term memory is the open recipe card on the counter right now: you can read every word at a glance, but the moment you clear the counter for the next dish, it is gone. Long-term memory is the cookbook on the shelf. It holds far more than one card could, and it stays there for years, but you have to walk over, find the right page, and trust that the recipe has not gone out of date since it was printed. An agent works the same way. What it can see this instant lives in its short-term memory. What it has to go fetch from a database or a file lives in its long-term memory. Knowing which goes where, and what each forgets, is most of the job.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent memory is one of the most muddled topics in interviews, because the word covers two systems that share almost nothing. One lives inside the model's prompt and the other lives outside it in a database. Conflating them is the single most common error, and it hides the fact that each has a completely different failure mode.
The clean way to reason about it is by three properties: where the information physically sits, how long it survives, and how the agent gets to it. Short-term memory and long-term memory land on opposite ends of all three. Getting this split right is what lets you say sensibly what to write where, what to forget, and what can break.
A useful framing borrowed from operating systems is to treat short-term memory as RAM and long-term memory as disk. RAM is fast and directly addressable but small and volatile; disk is large and durable but reached through an explicit read. The agent runtime, like an operating system, is responsible for paging the right facts between the two. The rest of this explanation walks through each tier, the policy that governs movement between them, and the failure each tier is prone to.
Short-term memory: the context window
Short-term memory is the context window. It is the running prompt the model sees on the current turn: the original goal, the recent tool calls and their results, and any scratchpad notes the agent is keeping for this task. Nothing has to be fetched, because every token is already in front of the model. That makes access effectively free in latency terms.
The defining traits are speed and impermanence. It is fast because there is no retrieval round trip and no external system in the path. But it is bounded by the model's context limit, and it is wiped the moment the session ends. There is no operation that persists it on its own. If you want any of it to survive, you have to copy it out to a store.
This is why short-term memory maps cleanly onto the idea of a working set. It holds exactly what the agent needs to reason about right now, and not a token more if you manage it well. The discipline is keeping that working set small enough to fit while still holding everything the current step depends on.
It is worth being precise about what counts as short-term memory, because larger context windows tempt people into thinking the problem has gone away. A million-token window is still short-term memory. It is still volatile and still wiped at session end. A bigger window raises the ceiling on how long a single task can run before overflow, but it does nothing for persistence across sessions, and it does not make the working set free. Long contexts also degrade in quality toward the middle, the so-called lost-in-the-middle effect, so even when a fact technically still fits, the model may not attend to it. Treating window size as a substitute for a real memory architecture is the trap; the window is a fast cache, not a database.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- ChatGPT's memory feature persists user facts to a long-term store across chats, while each conversation's running context is the short-term window.
- LangGraph separates a per-session checkpointer for short-term thread state from a long-term store keyed by user id that survives across threads.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you keep an agent's long-term memory from serving stale facts on retrieval?
Stamp every record with a write time and a source, attach a time to live where the fact is volatile, and re-validate or down-rank aged hits at read time. For high-stakes facts, prefer a live lookup over the cached vector and surface the age to the model so it can hedge.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling the vector store the agent's only memory. The context window is itself short-term memory, and conflating the two hides the distinct overflow and staleness failure modes.
60 second bullets to scan on the way to the call.
State that short-term memory is the context window for the current session.
State that long-term memory is an external store reached by retrieval.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.