Describe the three tier memory hierarchy used by modern LLM agent systems

Flashcard·Easy·4.0 · 0·~30s·Asked atInfosys Mu Sigma Stability Ai

Attempt it

Working memory: the active scratchpad for the current turn, partial reasoning, intermediate tool outputs, plans. Discarded at turn end. Short-term / session memory: the running conversation history for the current session, often summarized as it grows. Long-term memory: durable facts surviving across sessions, typically split into episodic (specific past events the user mentioned) and semantic (distilled preferences and stable facts). The hierarchy mirrors the cognitive-science taxonomy and maps onto the read-write paths most frameworks expose.

Your answer

Modern LLM agents are usually described as having three memory tiers, each with a different lifetime. **Working memory** is the scratchpad for the current turn. It holds partial reasoning, intermediate tool outputs, and the agent's current plan. Everything in working memory is discarded at the end of the turn, it is not persisted anywhere. This is the closest analog to human working memory. **Short-term memory** is the conversation history for the current session. It is the running record of user-assistant turns. As the session grows, short-term memory typically gets summarized, older turns get compressed into a rolling summary so the budget does not blow up. When the session ends, short-term memory ends with it (unless something gets promoted to long-term). **Long-term memory** is the durable layer that survives across sessions. It splits into two sub-types: **episodic memory** holds specific past events the user mentioned ('last Tuesday the user asked about flight bookings'), and **semantic memory** holds distilled facts and preferences ('the user prefers concise answers,' 'the user works in Python'). These persist in an external store (Mem0, Letta, Zep) and get retrieved on future sessions. The three-tier structure mirrors the cognitive-science taxonomy and lines up with the read-write paths most agent frameworks expose.

The three tiers are **working memory**, **short-term (session) memory**, and **long-term memory**, distinguished primarily by lifetime and write semantics. **Working memory** lives for the duration of a single turn or single tool-call sequence. It contains the agent's intermediate scratchpad, partial reasoning, tool outputs queued for the next step, the current plan, sub-goal tracking. Nothing in working memory survives the turn boundary; it is implicit in the assembled prompt for the current step and then discarded. **Short-term memory** is the session-scoped conversation history. It holds the running record of user-assistant turns, sometimes plus the working-memory snapshots from prior turns. As the session lengthens, the older portions are typically compressed into a rolling summary (LangChain `ConversationSummaryBufferMemory`, LlamaIndex chat memory, Anthropic's recommended compaction pattern). When the session terminates, short-term memory is discarded unless something is explicitly promoted to long-term. **Long-term memory** persists across sessions in an external store. It splits along the cognitive-science distinction: **episodic memory** holds specific events from past sessions ('on 2026-03-15 the user asked about JFK to SFO flights' or 'the user mentioned their daughter is in Berkeley'); **semantic memory** holds distilled stable facts and preferences ('the user prefers Python over JavaScript,' 'the user is allergic to peanuts'). The 2026 stack treats these as distinct stores: Mem0 specializes in semantic memory with extraction pipelines; Zep maintains a temporal knowledge graph with episodic facts; Letta (formerly MemGPT) gives the model memory-management tools to write its own long-term store. Each tier has different read and write paths, different update frequencies, and different failure modes. Mixing them up, for example treating session history as if it were long-term, is a common production bug.

TL;DR

Working memory (scratchpad, dies at turn end), short-term memory (session history, dies at session end), and long-term memory (durable facts across sessions, split into episodic and semantic).

Memory aid

Easy to grasp

Picture how you remember things during a conversation with a friend. While they are asking a question, you hold partial thoughts in your head: that is working memory, and it disappears when you respond. Across the whole conversation you remember what was already said: that is short-term memory, and it fades after the conversation ends. Across years of friendship you remember things they have told you: their job, their dog's name, the time they broke their leg skiing. That is long-term memory, and it survives across visits. Some of those long-term memories are specific events ('the ski accident'), others are settled facts about who they are ('they hate coriander'). LLM agents are built with the same three-layer structure.

Key concepts

Context Engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The three-tier memory hierarchy is the 2026 standard mental model for thinking about agent state. It borrows the cognitive-science taxonomy (working, short-term, long-term, with long-term split into episodic and semantic) and maps it onto the read-write paths an agent framework exposes. The point of the model is not the names, it is that each tier has a different lifetime, and confusing the lifetimes is one of the most common production bugs.

This deep dive walks each tier in order, names the failure modes that arise when the boundaries blur, and maps the tiers onto the 2026 framework landscape (Mem0, Letta, Zep, LangGraph).

Mental model: three lifetimes, turn, session, cross-session. Each lifetime gets its own store, its own write policy, and its own failure modes.

Working memory: the per-turn scratchpad

What it holds

Working memory is the active scratchpad for the current turn or single tool-call iteration. It contains the agent's partial reasoning, intermediate tool outputs that have not yet been folded into the conversation, the current plan, and the immediate sub-goal. In a ReAct-style agent, working memory is the running 'Thought / Action / Observation' chain inside one user request.

Lifetime

Working memory does not survive the turn boundary. When the agent finishes its work and returns a response to the user, the working memory is implicitly discarded. Nothing about it is persisted unless it is explicitly promoted to short-term memory (typically by being included in the assistant's response message).

Where it lives mechanically

In practice, working memory often lives in the assembled prompt itself, as a scratchpad block placed near the recency slot. The agent reads the scratchpad, appends new content, and the next forward pass sees the updated version. There is rarely a separate 'working memory store' on disk; it is ephemeral by design.

Failure modes

The main failure mode is treating working memory as persistent. A developer writes 'remember this for later' into the agent's scratchpad expecting it to survive, then is confused when the next session has no record. The fix is to explicitly promote: if something needs to survive, write it to long-term memory via a tool call before the turn ends.

Short-term memory: session-scoped conversation

Long-term memory: cross-session durability

How the tiers interact in a real agent

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Mem0 (2024-2026) maintains a semantic-memory store with extraction pipelines and update on contradiction policies.
Letta (formerly MemGPT) exposes memory-management tools so the agent can rewrite its own long-term store inside the prompt.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the write policy that decides when something gets promoted from short-term to long-term?

Three common policies: eager (extract facts every turn, expensive and noisy), lazy on signal (only when the user says 'remember this' or similar), and batch on session end (sweep the session, extract durable facts, write once). Production systems often combine batch on end with explicit user-triggered writes.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Conflating all 'memory' as one flat store, then being surprised when a fact written during a session is missing from the next session, different tiers have different lifetimes.

60 second bullets to scan on the way to the call.

Names and lifetimes of the three tiers
How session memory is compacted as it grows

Primary sources. Browse if you want the original framing.