How should a long running agent task persist state so it can resume after a crash or deployment?

MCQ·Medium·4.0 · 0·~1 min·Asked atDatarobot Jane Street Lambda Labs·Relevant atAnthropic

Attempt it

A long agent task is fragile in the obvious way. The worker can crash, the pod can be evicted, a deploy can roll the process, a timeout can trigger a kill. If the task state lives only in process memory, every one of these events loses the work. The right pattern is **checkpointing**. After every step that changes state, the agent writes a checkpoint to a durable store. The checkpoint contains everything needed to continue: the full message history, the current step index, the results of any tool calls, and the next planned action. The store has to be durable and shared. Postgres, Redis with persistence enabled, or S3 are all reasonable. The key is the task id, so any worker can look up any task. When a worker picks up a task, it loads the latest checkpoint and continues from there. The other options fail in different ways. In-memory state vanishes on any restart. Local disk dies when the pod is replaced and is not visible to other workers. Snapshotting the model weights is meaningless because the model is loaded by the runtime, not stored per task. Frameworks like LangGraph ship a checkpointer abstraction backed by Postgres or similar. In production you almost always use the framework's built-in checkpointer rather than rolling your own.

The pattern is **durable checkpointing keyed by task id**, written after every state-mutating step. The checkpoint contents are the minimum needed to resume cleanly: the full message history, the current step index, the results of tool calls already executed, and the next planned action. The store is something that survives any single worker dying: Postgres, Redis with append-only persistence, or S3 with object versioning. LangGraph's `Checkpointer` plus Postgres is the canonical reference implementation in 2026. On resume, a worker loads the checkpoint for the task id, reconstitutes the agent state, and continues from the next planned step. The same checkpoint key serves both the resume case (after a crash) and the hand-off case (a different worker picks up the task). There is no special crash-recovery code path; resume is just load and continue. The other options each fail a specific test. In-memory state does not survive the typical pod lifetime, which is minutes, not the 30 minutes a long task can take. Local disk survives a process restart but not a pod replacement and is not shared with other workers. Model-weight snapshots confuse model state (loaded by the runtime) with task state (the only thing that varies per request). For tasks longer than a typical pod lifetime, anything other than durable checkpointing is unsafe.

TL;DR

Checkpoint after every state-mutating step to a durable store keyed by task id, with full message history, step index, and tool results, so any worker can resume.

Memory aid

Easy to grasp

Imagine writing a long story in chapters. Every time you finish a chapter, you save the whole story so far to a folder on a server, not just on your laptop. If your laptop dies, you can pick up any other laptop, open the folder, see exactly where you left off, and write the next chapter. An agent task is the same. After every step it takes, it writes down the entire story so far, the step number it is on, what tools it called and what they returned, and what it plans to do next. The notes live in a database that survives any single computer crashing. Any worker can pick up any task by reading the notes and continuing.

Key concepts

Checkpointing Durable State LangGraph

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A long-running agent task is a hostile environment for in-memory state. The worker process can crash from an OOM, the pod can be evicted by the scheduler, a deploy can roll the pod, a timeout can kill the call, a network blip can disconnect a tool, an upstream service can drop a request mid-loop. Any of these events lose the entire task if the only place the state lives is RAM. The 30-minute task in the question is several typical pod lifetimes in a normal Kubernetes cluster, so the question is not whether the worker will die, but when.

The correct answer is durable checkpointing after every state-mutating step, keyed by task id, written to a store that outlives any single worker. The store of choice in 2026 is typically Postgres, which is what LangGraph's reference checkpointer uses. Other durable backends like Redis with append-only persistence or S3 with object versioning also work; the requirement is durability and visibility across workers, not the specific technology.

The rest of this explanation walks through what goes in a checkpoint, where you write it relative to tool calls, why each of the three wrong answers fails, and where durable workflow engines like Temporal and Inngest fit on top of plain checkpointing.

What a checkpoint must contain

A checkpoint is not just a transcript dump. It is the minimum state needed for any worker to load it and continue cleanly from the next step. Four pieces are non-negotiable.

The full message history comes first. The agent's reasoning depends on the prompt as it stands right now, including every observation it has seen so far, so the message list must be reconstructible verbatim. Truncating the history at resume time would change the model's behaviour on the next call.

The current step index is the cursor that says where the agent is in its loop. Resume needs to know whether the next thing to do is plan, call a tool, observe a tool result, or finalise an answer. Without an index, the resuming worker has to infer state from the messages, which is brittle.

The tool-call results that have already been executed are recorded so resume does not re-execute them. If the agent called a search tool at step 4 and the result is in the checkpoint, step 4 does not run again. This is the half of the idempotency contract that the checkpoint controls; the other half is in the tool design itself.

The next planned action is the agent's intent: which tool will be called next with which arguments, or which final answer is being assembled. Recording the intent is what makes resume restartable from a uniform state regardless of where in the loop the previous worker died.

With all four pieces, a checkpoint becomes a complete picture: not just what happened, but where the agent was about to go. Resume becomes load and continue with no special crash-recovery code path.

Where to write the checkpoint and the idempotency contract

Why the three wrong options fail

LangGraph, Temporal, and the durable-workflow landscape in 2026

Operational considerations: keys, retention, and observability

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LangGraph ships a `Checkpointer` interface with Postgres, SQLite, and in-memory backends; Postgres is the canonical production choice for long-running agent tasks.
Temporal's durable workflow engine is widely used as the orchestration layer for multi-step agent pipelines at scale, providing exactly-once semantics on tool calls.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhere exactly do you write the checkpoint: before the tool call, after the tool call, or both?

Both, with idempotency keys. Before the call, you record the intent so resume knows what was about to happen. After the call, you record the result so resume does not re-execute. If you only checkpoint after, a crash between intent and execution loses the planned action. If you only checkpoint before, idempotency is the only thing protecting against double execution on resume.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Holding agent state in process memory or on local disk, so any pod restart, deployment, or crash loses progress and forces the task to start over.

60 second bullets to scan on the way to the call.

Explain why in-memory state fails for any task longer than a pod lifetime.
Name the four things a checkpoint must contain: messages, step index, tool results, next action.

Primary sources. Browse if you want the original framing.