Explain the LLM vs runtime split in an agent and why the distinction matters for debugging

Attempt it

An agent system has two major components: the LLM and the runtime. Describe the responsibility of each, then explain why confusing them leads to incorrect debugging assumptions.

Free · 2 AI evals / day

Quick reference

TL;DR

The LLM only decides and emits text; the runtime parses that text, executes tools, and feeds results back. Knowing who does what tells you where a bug actually lives.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a brilliant chess coach who is locked in a soundproof booth. The coach can see the board through glass and can shout out moves, but the coach can never touch a piece. A second person, the assistant, stands at the board, hears the move, moves the piece, and then describes the new board back through the glass. The coach is the brain that decides. The assistant is the hands that act. If a piece ends up in the wrong square, you have to ask the right question. Did the coach call a bad move, or did the assistant hear it wrong and move the wrong piece? An agent works the same way. The model is the coach in the booth, and the runtime is the assistant at the board.

Key concepts

Agent Runtime Tool Calling Agent Observability

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Define the two layers, state that the model only emits text while the runtime executes, explain that injecting the result closes the loop, then map failure types to the layer that owns them and close with why per-step tracing is the only way to localize a bug.

Real-world examples

Real products, models, and research that use this idea.

The Anthropic SDK returns a tool_use content block from Claude Opus 4.7; your code, not the model, must execute it and post back a tool_result block to close the loop.
OpenAI function calling returns a tool_calls array as JSON text. The model never runs the function; the developer's runtime parses the arguments and dispatches the real call.
LangGraph models the split explicitly: the LLM node emits the decision and a separate ToolNode executes it, so traces show exactly which layer produced a failure.
LangSmith and Langfuse trace each step separately, recording the model's emitted tool call alongside the runtime's observed result so engineers can localize a bug to one layer.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf the model emits a tool call but the result never appears in the next turn, where do you look first?

Sign in or go Pro to see the approach for this follow-up.

QHow does the split change when a model supports parallel tool calls in one turn?

Sign in or go Pro to see the approach for this follow-up.

QWhy can prompt injection in a tool result compromise the whole loop despite the split?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying the model 'called' a tool. The model only emits a tool-call object as text. The runtime parses it and decides whether to execute it, so a silent failure can hide there.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.