Where does an agent keep short term state during a single task?
The agent's per-task working state (conversation, tool results, plans, scratchpad). It lives in the prompt or runtime state, is bounded by the context window, and resets at task end.
Imagine someone solving a puzzle at a table. They keep a little notepad next to them with arrows and crossed-out guesses, the picture of the box, and a few tiles already placed. When the puzzle is done, they throw the notepad away and the next puzzle starts with a fresh notepad. Short-term memory is that notepad for an agent. It holds everything the agent needs to remember while working on this one task: what was said earlier in the conversation, what the last tool returned, what the agent is planning to do next. When the task ends, the notepad is gone.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Short-term memory in an agent is the working state that exists within one task run. It holds the conversation with the user, the sequence of tool calls and their results, intermediate plans the agent has formed, and any scratchpad notes used during reasoning. When the task ends, that state is normally discarded and the next task starts fresh.
The core idea is easy to misstate because the word 'memory' carries human connotations the model does not actually have. The language model itself is stateless. Each call is an independent forward pass. The model does not 'remember' anything from the previous call; it sees a prompt, produces tokens, and forgets immediately. Short-term memory is therefore not a property of the model. It is a property of the runtime: the harness that calls the model many times during one task and assembles the prompt that gives each call the illusion of continuity.
This explanation unpacks what lives in short-term memory, where it physically resides, what bounds its useful size, and how it differs from long-term memory. By the end you should be able to point at any byte the agent sees and say with confidence whether it counts as short-term memory, long-term memory, or neither.
What short-term memory contains
The contents of short-term memory fall into a small number of categories that recur across every agent framework.
The first is the conversation history with the user. The user message that started the task, plus any clarifying back-and-forth that has happened since, sit in short-term memory throughout the task.
The second is the sequence of tool calls and their results. Each call the agent has made, with its arguments, and each response the tool returned, are recorded in the state. This is the part that grows fastest in long-running agents, because each tool result can be substantial.
The third is intermediate planning. Many agent patterns explicitly produce a plan or a sub-goal early on and reference it on subsequent steps. The plan sits in short-term memory so each step can read it.
The fourth is scratchpad notes. Some agents emit reflections, observations, or step-by-step reasoning that the next step can read. These do not necessarily go to the user; they are intermediate state used internally.
All four categories share one property: they exist for the duration of the current task and are normally discarded when the task ends. They are also all that the model gets to see on any given call, because the model is stateless and the runtime assembles the prompt from the state on every step.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangGraph's state graph keeps a typed state object per task; each node mutates the state and the runtime renders it into the prompt on each LLM call.
- The OpenAI Agents SDK and Anthropic SDK both treat short-term memory as a per-session message list passed in on every call, with the runtime appending tool calls and results.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf frontier models have million-token context windows, why does effective short-term memory feel much smaller in practice?
Attention is not uniform. Lost-in-the-middle and salience competition cause quality to drop on long contexts well before the capacity limit. Effective memory is the band the model attends to reliably, which is typically a fraction of the nominal window.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing short-term memory is something the model 'has' the way humans have working memory. The model is stateless; short-term memory is just the prompt and runtime state assembled fresh on every call.
60 second bullets to scan on the way to the call.
Define short-term memory as the agent's per-task working state.
List what it contains: conversation, tool calls and results, plans, scratchpad notes.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.