Spot the error in this agent runtime's tool-call handling code

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

The runtime trusts the model's tool call blindly. It never checks the tool exists, so a hallucinated name returns None and crashes on .execute instead of becoming a recoverable error observation.

Memory aid

Easy to grasp

Imagine a waiter who takes any order a customer shouts and runs to the kitchen to make it, even if the dish is not on the menu. When the kitchen has no recipe, the waiter freezes mid-step and the whole restaurant stops. A better waiter checks the menu first. If the dish does not exist, they calmly tell the customer it is not available and ask them to pick something real. The customer can then choose again. This code is the bad waiter. It takes whatever tool name the model invents and tries to run it. When the name is not on the menu, the lookup hands back nothing, and the next line breaks because you cannot run nothing. The fix is to check the menu before cooking.

Key concepts

Tool Schema Validation Hallucinated Tool Calls Error as Observation

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

State the bug precisely, that an unguarded execute on a None lookup crashes the loop. Then explain why a tool call is untrusted input, lay out the validate then reject and reprompt pattern over both name and schema, and close with error message quality and the try block around execution itself.

Real-world examples

Real products, models, and research that use this idea.

The Anthropic SDK and OpenAI tool use both return a structured tool result message on failure, so a bad call from Claude Opus 4.7 or GPT-5.5 becomes an observation the model reasons about rather than a crash.
LangGraph's prebuilt ReAct agent routes an unknown tool name back as a ToolMessage error, keeping the state graph alive so the model can pick a valid tool on the next node.
Pydantic AI validates tool arguments against a typed model before execution, and a validation error is fed back to the model as a retry prompt automatically.
MCP servers declare a JSON schema per tool, and a compliant host validates the model's arguments against that schema before dispatching the call to the server.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design the error observation so the model is most likely to self correct on the next turn?

QWhen does validation alone fail to protect you, and what else must the runtime enforce on a tool call?

QHow would you stop a model that keeps emitting the same invalid tool call after each rejection?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming the model only ever names tools that exist. Models hallucinate function names and malformed arguments, so the runtime must validate every call against the registry and schema before executing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.