Describe the three-tier memory hierarchy used by modern LLM agent systems

Flashcard·Easy·4.0 · 0·~30s·Asked atAnthropic OpenAI Perplexity

Attempt it

Working memory: the active scratchpad for the current turn, partial reasoning, intermediate tool outputs, plans. Discarded at turn end. Short-term / session memory: the running conversation history for the current session, often summarized as it grows. Long-term memory: durable facts surviving across sessions, typically split into episodic (specific past events the user mentioned) and semantic (distilled preferences and stable facts). The hierarchy mirrors the cognitive-science taxonomy and maps onto the read-write paths most frameworks expose.

Your answer

Modern LLM agents are usually described as having three memory tiers, each with a different lifetime. **Working memory** is the scratchpad for the current turn. It holds partial reasoning, intermediate tool outputs, and the agent's current plan. Everything in working memory is discarded at the end of the turn, it is not persisted anywhere. This is the closest analog to human working memory. **Short-term memory** is the conversation history for the current session. It is the running record of user-assistant turns. As the session grows, short-term memory typically gets summarized, older turns get compressed into a rolling summary so the budget does not blow up. When the session ends, short-term memory ends with it (unless something gets promoted to long-term). **Long-term memory** is the durable layer that survives across sessions. It splits into two sub-types: **episodic memory** holds specific past events the user mentioned ('last Tuesday the user asked about flight bookings'), and **semantic memory** holds distilled facts and preferences ('the user prefers concise answers,' 'the user works in Python'). These persist in an external store (Mem0, Letta, Zep) and get retrieved on future sessions. The three-tier structure mirrors the cognitive-science taxonomy and lines up with the read-write paths most agent frameworks expose.

The three tiers are **working memory**, **short-term (session) memory**, and **long-term memory**, distinguished primarily by lifetime and write semantics. **Working memory** lives for the duration of a single turn or single tool-call sequence. It contains the agent's intermediate scratchpad, partial reasoning, tool outputs queued for the next step, the current plan, sub-goal tracking. Nothing in working memory survives the turn boundary; it is implicit in the assembled prompt for the current step and then discarded. **Short-term memory** is the session-scoped conversation history. It holds the running record of user-assistant turns, sometimes plus the working-memory snapshots from prior turns. As the session lengthens, the older portions are typically compressed into a rolling summary (LangChain `ConversationSummaryBufferMemory`, LlamaIndex chat memory, Anthropic's recommended compaction pattern). When the session terminates, short-term memory is discarded unless something is explicitly promoted to long-term. **Long-term memory** persists across sessions in an external store. It splits along the cognitive-science distinction: **episodic memory** holds specific events from past sessions ('on 2026-03-15 the user asked about JFK to SFO flights' or 'the user mentioned their daughter is in Berkeley'); **semantic memory** holds distilled stable facts and preferences ('the user prefers Python over JavaScript,' 'the user is allergic to peanuts'). The 2026 stack treats these as distinct stores: Mem0 specializes in semantic memory with extraction pipelines; Zep maintains a temporal knowledge graph with episodic facts; Letta (formerly MemGPT) gives the model memory-management tools to write its own long-term store. Each tier has different read and write paths, different update frequencies, and different failure modes. Mixing them up, for example treating session history as if it were long-term, is a common production bug.

Quick reference

TL;DR

Working memory (scratchpad, dies at turn end), short-term memory (session history, dies at session end), and long-term memory (durable facts across sessions, split into episodic and semantic).

Memory aid

Easy to grasp

Picture how you remember things during a conversation with a friend. While they are asking a question, you hold partial thoughts in your head: that is working memory, and it disappears when you respond. Across the whole conversation you remember what was already said: that is short-term memory, and it fades after the conversation ends. Across years of friendship you remember things they have told you: their job, their dog's name, the time they broke their leg skiing. That is long-term memory, and it survives across visits. Some of those long-term memories are specific events ('the ski accident'), others are settled facts about who they are ('they hate coriander'). LLM agents are built with the same three-layer structure.

Key concepts

Context Engineering

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 to 4 min: name the three tiers by lifetime, explain the episodic vs semantic split, and walk through how 2026 frameworks specialize in each tier.

Real-world examples

Real products, models, and research that use this idea.

Mem0 (2024-2026) maintains a semantic-memory store with extraction pipelines and update on contradiction policies.
Letta (formerly MemGPT) exposes memory-management tools so the agent can rewrite its own long-term store inside the prompt.
Zep maintains a temporal knowledge graph storing episodic facts with timestamps, supporting time-aware retrieval.
LangGraph exposes typed state that flows through nodes and supports checkpointing for short-term and long-term tiers via different stores.
ChatGPT's memory feature (rolled out 2024-2025) is a semantic-memory implementation: user-stated preferences and durable facts surface across sessions.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the write policy that decides when something gets promoted from short-term to long-term?

QHow does the agent decide what to retrieve from long-term memory at the start of a new session?

QHow do episodic and semantic memory interact when they contradict?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Conflating all 'memory' as one flat store, then being surprised when a fact written during a session is missing from the next session, different tiers have different lifetimes.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.