Flashcard: what is an 'agent runtime' and what does it do?

Flashcard·Easy·4.0 · 0·~30s·Asked atGoldman Sachs Stability Ai Turbopuffer

Attempt it

Your answer

**The agent runtime** is the code that sits around the language model and turns it into an agent. The model itself only produces text and forgets everything the moment a call ends. Every other operational concern lives in the runtime. Concretely, the runtime does five things. It runs the loop: call the model, parse the response, execute any tool calls, append the results, and call the model again. It owns the tool registry: the set of functions the model can ask for, plus their schemas. It manages state: the conversation history, intermediate results, and any persistent memory between turns. It enforces stopping conditions: the step limit, the cost cap, the timeout, the final-answer detection. It emits traces and logs so you can debug what the agent actually did. The runtime is the part you write or import from a framework. LangGraph, CrewAI, AutoGen, and the OpenAI Agents SDK are popular agent runtimes in 2026. Each gives you a slightly different way to express the loop (state graph, role-play crew, multi-agent conversation, function-calling SDK), but underneath they all do the same five things. The phrase to remember is: the model is the brain; the runtime is the body. Without a runtime, an LLM call is just a chat completion.

**The agent runtime** is the orchestration layer around the LLM. It owns the loop (call model, parse output, execute tool, append result, repeat), the tool registry, the conversation state and memory, the stopping conditions, and the observability hooks. This is the deterministic code half of an agent. The model is non-deterministic and stateless across calls; the runtime is the stable code you wrote that connects model decisions to real effects. When the model emits a tool-use block, the runtime parses it, validates the arguments against the tool's schema, executes the function, captures the result (including errors), and sends it back as the next observation. When the model emits a final answer with no tool call, the runtime detects that and exits the loop. Every operational lever lives in the runtime. Step budgets, cost caps, wall-clock timeouts, retry policies, idempotency keys, permission checks, and trace spans are all runtime concerns. The model cannot enforce any of these on itself because it has no view of the budget and no ability to refuse a call. Frameworks in 2026 each pick a different topology. LangGraph models the loop as a typed state graph. CrewAI models a crew of role-playing agents collaborating. AutoGen models agents as participants in a multi-party conversation. The OpenAI Agents SDK and Anthropic's MCP-aware tooling expose tool calling as a primitive and let you build the loop directly. Underneath, all four are runtimes implementing the same observe reason act repeat pattern.

TL;DR

The agent runtime is the deterministic code around the LLM that drives the loop, executes tools, manages state, enforces budgets, and emits traces. LangGraph, CrewAI, AutoGen, and the OpenAI Agents SDK are runtimes.

Memory aid

Easy to grasp

Picture a chess engine paired with a robot arm. The engine picks the move, but the arm is what actually slides the piece across the board, hits the clock, and writes down the move on the scoresheet. The engine alone is just numbers on a screen; the arm alone has no idea what to play. The agent runtime is the robot arm and the scoresheet and the clock and the rulebook, all rolled into one. The LLM only ever says what should happen next. The runtime is the part that actually makes it happen, keeps notes, watches the time, and decides when the game is over. Frameworks like LangGraph and CrewAI ship as ready-made runtimes so you do not have to build the arm yourself.

Key concepts

Agent Runtime Orchestration Agent Loop

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The agent runtime is the most under-appreciated half of an LLM agent. People talk about the model that powers the agent: Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro. But every reliable agent in production also has a runtime around the model, and most of the engineering work that makes the agent actually work happens there.

This deep dive defines what a runtime is, walks through the five responsibilities it owns, contrasts the popular 2026 frameworks by how they express those responsibilities, and explains why putting work in the runtime versus the prompt is a recurring engineering tradeoff. By the end, the line between the model's job and the runtime's job should be clear enough to apply when reading any agent codebase or framework documentation.

What the runtime actually is

The runtime is the deterministic code around the LLM that turns model calls into agent behavior. The model alone produces text and forgets between calls. The runtime is everything that makes those text outputs add up to a multi-step task being completed: the loop, the tool execution, the state, the budgets, and the traces.

Mechanically, the runtime is a function (or service) you call with a task description. Internally it builds a prompt, calls the model, inspects the response, decides what to do next, and repeats until a stopping condition fires. You can write a 50-line runtime by hand for a toy use case; production runtimes are thousands of lines and often a separate service with its own database for persistent memory.

The phrase orchestration layer captures the runtime's role. It orchestrates the model, the tool functions, the memory store, and the observability backend into a coherent system. The model is one component among several, and the runtime is what connects them in the right order with the right safety controls.

Responsibility one: the loop

Responsibility two: state and memory

Responsibility three: tools, trust, and stopping conditions

The 2026 framework landscape

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LangGraph: the runtime is a typed state graph; you declare nodes for the model and each tool, and the framework drives the loop with explicit edges and recursion limits.
CrewAI: the runtime coordinates multiple role-playing agents (planner, researcher, writer) with built-in delegation and shared memory.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf the model is stateless across calls, how does an agent remember what it did three turns ago?

The runtime accumulates the transcript and rebuilds the context window each turn, optionally summarizing or storing older observations in a side store and re-injecting only relevant slices.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Calling the runtime 'just glue' or 'plumbing'. The runtime owns budgets, state, retries, validation, and observability; most agent reliability work happens in the harness, not the prompt.

60 second bullets to scan on the way to the call.

List the five things a runtime owns: loop, tool registry, state, stopping conditions, observability.
Explain why the model cannot enforce its own budgets or remember state across calls.

Primary sources. Browse if you want the original framing.