Define a hallucinated tool call in the context of LLM agents. Name three defensive layers a production runtime can apply to handle them.
A hallucinated tool call names a tool that does not exist or passes arguments that break its schema. Defend with schema validation, constrained output, and capped retry on error.
Imagine a new hire who is told the office has a printer, a scanner, and a coffee machine. Eager to help, they confidently say they will use the fax machine, which the office does not own. Or they try to use the printer but feed it a banana instead of paper. That is a hallucinated tool call. The model is sure it knows what tools exist and how to use them, but it is wrong. A good runtime acts like a careful receptionist. It checks every request against the real list of machines, rejects nonsense politely, and explains the mistake so the hire can try again. It also limits how many times they can keep guessing, so nobody wastes the whole afternoon on a fax machine that was never there.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A hallucinated tool call is one of the most common failure modes in production agents, and it is also one of the most preventable. It occurs when the model emits an action the runtime cannot honour: a function name that is not in the registered tool set, or an argument object that violates the tool's declared schema. The model is not aware it is wrong. It produces the invalid call with the same fluent confidence it produces a correct one, because at generation time it has no way to check its own output against the real world.
The key framing for an interview is that this is a runtime responsibility, not a prompting problem. You cannot fully prompt a probabilistic model out of occasionally inventing a tool. What you can do is build a boundary that no invalid call can cross, prevent most invalid calls from ever being generated, and turn the rest into feedback the model can recover from.
The reason this matters so much in agents specifically, rather than in plain chat, is blast radius. A hallucinated tool call is not just a wrong sentence. It is an attempt to take an action with real side effects: query a database, write a file, charge a card, send an email. If the runtime executes a malformed or invented call without checking it, the failure escapes the model and becomes a production incident. So the defense is not about politeness, it is about never letting an unvalidated action reach a real system.
What exactly counts as a hallucinated tool call
There are two distinct shapes, and a good answer names both. The first is a non-existent function name. The model emits a call to a tool like fetch_weather when the registry only contains get_forecast. The name is plausible, even reasonable, but it does not exist. The runtime has nothing to dispatch to. A close cousin is a real name with the wrong casing or namespace, which is still a registry miss.
The second shape is invalid arguments to a real tool. Here the function name is correct, but the arguments break the schema. The model passes a string where an integer is required, omits a required field, invents an enum value the tool does not accept, or supplies a date in the wrong format. The tool exists, but the call cannot be executed safely. This second shape is sneakier than the first, because a name miss fails loudly while a subtle type or range error can pass a weak check and corrupt downstream state.
It is worth separating hallucinated calls from two neighbours interviewers like to probe. A correctly formed call to the wrong tool for the task is a reasoning error, not a hallucination, because the call itself is valid. A correctly formed call with semantically wrong but schema-valid arguments, such as a real but incorrect user id, is also not a schema hallucination. The defenses here target structural validity. Semantic correctness needs separate guardrails like verification or human approval.
Both shapes are confident errors. The model has no runtime feedback at generation time, so it cannot tell a valid call from an invalid one. That is precisely why the runtime, not the model, has to be the source of truth about what is callable.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI and Anthropic tool use APIs accept a JSON schema per tool and, with strict structured output, constrain the model so it cannot emit an unregistered function name.
- LangGraph and the Anthropic SDK validate tool arguments against the declared schema at the boundary and surface the validation error back into the agent state as an observation.
What an interviewer would ask next. Try answering before peeking at the approach.
QConstrained decoding guarantees a valid function name and schema. Why still validate at the runtime boundary?
Talk about defense in depth, providers that only soft-enforce schemas, semantic argument checks the schema cannot express, and calls from a model or backend you do not fully control.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating a hallucinated call as a model bug to fix with prompting alone. The real fix is a runtime validation boundary plus constrained output, because no prompt fully stops the model from inventing a tool.
60 second bullets to scan on the way to the call.
Define a hallucinated tool call precisely, covering both wrong name and wrong arguments.
Name the main causes: weak schema grounding, too many tools, long context attention decay.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.