Explain the difference between short-term and long-term memory in an agent system. Give one failure mode that is unique to each type.
Free · 2 AI evals / day
An agent has two kinds of **memory**, and they sit in different places. The first is short-term memory. This is just the context window, the running prompt that holds the current task, the recent tool results, and the conversation so far. It is fast to read because it is already in front of the model, and it needs no extra system. But it is small and it disappears when the session ends.
The second is long-term memory. This is an external store, usually a vector database, a regular database, or a set of files. The agent does not see it automatically. It has to run a retrieval step to pull the right pieces back into the context window when they are needed.
The simplest way to remember the split is by location. Short-term memory is what the model can see right now. Long-term memory is what it has to go fetch.
Each kind fails in its own way. Short-term memory overflows: once the task grows past the window, early steps fall out and the agent forgets what it already did. Long-term memory goes stale: a fact saved months ago can be wrong today, and the agent has no built-in way to know it has aged.
Agent memory splits by where the information lives and how the agent reaches it. **Short-term memory** is the context window. It holds the current task, recent tool calls, and the working scratchpad for this session.
It is fast because no retrieval is involved, and it needs no external infrastructure. The cost is that it is bounded by the context limit and is wiped when the session ends.
Long-term memory is an external persistent store: a vector database, a relational database, a key-value store, or plain files. It survives across sessions and is effectively unbounded. The agent reaches it through an explicit retrieval step that pulls relevant slices back into context on demand.
The write policy follows from this. Put transient, this-task state in short-term memory. Promote durable facts, user preferences, and learned outcomes to long-term storage.
Each type has a signature failure. Short-term memory suffers context overflow: when history exceeds the window, early steps silently drop and reasoning degrades. Long-term memory suffers stale retrieval: a fact true at write time is served today as if still current. The agent has no flag that the data has aged, so it cites the old value with full confidence.
Quick reference
TL;DR
Short-term memory is the live context window for the current task; long-term memory is an external store retrieved as needed. Each fails differently: overflow versus staleness.
Imagine you are cooking from a recipe. Short-term memory is the open recipe card on the counter right now: you can read every word at a glance, but the moment you clear the counter for the next dish, it is gone. Long-term memory is the cookbook on the shelf. It holds far more than one card could, and it stays there for years, but you have to walk over, find the right page, and trust that the recipe has not gone out of date since it was printed. An agent works the same way. What it can see this instant lives in its short-term memory. What it has to go fetch from a database or a file lives in its long-term memory. Knowing which goes where, and what each forgets, is most of the job.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A single-turn task whose data fits comfortably inside one context window. Adding a long-term store buys nothing and only introduces retrieval latency and a staleness surface.
Facts that change every few seconds, such as live prices or sensor readings. A persisted store goes stale faster than it can be refreshed, so a direct live query beats stored memory.
Highly regulated data where retention itself is a liability. Persisting user content long-term creates compliance exposure that a session-scoped window avoids entirely.
Short bounded conversations where the whole history always fits in the window. Overflow never triggers, so summarisation and an external store add complexity for no benefit.
Quick scan · 2–4 min
Everything important, quickly.
Define each tier by location, persistence, and access path, give the write and forget policy that decides what goes where, then name the two distinct failure modes, overflow for short-term and staleness for long-term, and close with the mitigation each one demands.
Real-world examples
Real products, models, and research that use this idea.
1ChatGPT's memory feature persists user facts to a long-term store across chats, while each conversation's running context is the short-term window.
2LangGraph separates a per-session checkpointer for short-term thread state from a long-term store keyed by user id that survives across threads.
3Claude's projects and memory let durable instructions persist long-term, while the active conversation lives in the context window for that session.
4MemGPT and the Letta framework explicitly page facts between an in-context working memory and an external archival store, treating the window like RAM and the store like disk.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you keep an agent's long-term memory from serving stale facts on retrieval?
Sign in or go Pro to see the approach for this follow-up.
QWhere exactly does the boundary sit between short-term and long-term memory in a framework like LangGraph or MemGPT?
Sign in or go Pro to see the approach for this follow-up.
QWhat write policy decides whether a given observation goes into short-term or long-term memory?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Calling the vector store the agent's only memory. The context window is itself short-term memory, and conflating the two hides the distinct overflow and staleness failure modes.