How does function calling enable the observe plan act agent loop?

MCQ·Easy·4.0 · 0·~1 min·Asked atDeloitte Dust Hebbia·Relevant atLangChain OpenAI

Attempt it

**Function calling** is a provider-side feature in modern LLM APIs (OpenAI tools, Anthropic tool_use, Google function_declarations) where the model emits a structured tool-call JSON instead of plain text when it decides to invoke a function. The flow is straightforward. You declare available functions to the API with their schemas. When the model decides to call one, the API returns a structured tool-call object: the function name plus arguments as JSON. The runtime parses that JSON and executes the corresponding function. The result is sent back as a tool-result message, and the model continues. The key property is that the structured output is guaranteed at the token-sampling layer. The model is not writing JSON-like text that you then parse with regex; the API enforces well-formedness during sampling, so parsing is deterministic. Why this matters for agent loops. Every multi-step pattern (ReAct, Plan-and-Execute, agentic RAG) has an Action step where the model picks a tool. With function calling, that step emits clean structured JSON every time, and the runtime can route to the right tool with confidence. Function calling is the substrate that makes tool-using agents practical at scale.

**Function calling** is a provider-side API feature where the LLM emits a structured tool-call JSON object instead of plain text when it decides to invoke a function. It is provided by OpenAI (as 'tools'), Anthropic (as 'tool_use' content blocks), Google (as 'function_declarations'), and most modern open-weight models (Llama 4, Mistral Large 3, Qwen 3.5, DeepSeek V4). The flow has four steps. The developer declares available functions to the API with their JSON Schema. During inference, the model decides whether to emit text or a tool call. When it emits a tool call, the API returns a structured object with the function name and arguments. The runtime parses the object, executes the function, and sends the result back as a tool-result message. The load-bearing property is that the structure is guaranteed at the token-sampling layer. Providers enforce well-formedness during sampling (via constrained decoding or grammar-aware sampling), so the JSON is always parseable. Why this matters for agent loops. Every multi-step pattern (ReAct, Plan-and-Execute, agentic RAG) has an Action step where the model picks a tool. Before function calling, this step relied on parsing free text with regex, which was fragile. Function calling makes the Action step deterministic and reliable, which is what made tool-using agents practical at production scale. Function calling is the inference-layer mechanism; MCP is the complementary system-layer protocol. Together they form the substrate for modern agents.

TL;DR

Function calling is a provider-side feature where the model emits structured tool-call JSON instead of plain text. The deterministic parseability is what makes tool-using agent loops practical at scale.

Memory aid

Easy to grasp

Imagine asking an assistant to look something up. One way: they write a sentence saying 'please search for X' and you read the sentence and try to figure out what they meant. The other way: they hand you a small form with the search field filled in, so there is no ambiguity about what they want. Function calling is the second way. Modern LLM APIs let the model produce a structured tool-call form (a JSON object with the function name and arguments) instead of a free-text request. The runtime can read the form directly and call the right function. No guessing, no regex. This deterministic shape is what makes agent loops reliable enough to ship in production.

Key concepts

Function Calling Tool Use Structured Output Agent Loop

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Function calling is the provider-side API feature where an LLM emits a structured tool-call JSON object instead of plain text when it decides to invoke a function. The output structure is guaranteed by the API at the token-sampling layer, not by the prompt or by regex parsing of free text.

This feature is the foundation that makes tool-using agent loops practical at production scale. Every multi-step agent pattern (ReAct, Plan-and-Execute, agentic RAG, Reflexion) has an Action step where the model picks a tool, and function calling is what makes that step reliable across many turns.

This explanation walks through what function calling is mechanically, why it differs from prompt-engineered tool calls, how it interacts with MCP, and what it does not solve.

The mechanism: structured output guaranteed at sampling

Function calling is implemented in the API layer of modern LLM providers. The developer declares the available functions to the API along with their schemas (typically JSON Schema). During inference, the model can choose between emitting plain text and emitting a tool call. When it emits a tool call, the API returns a structured object with the function name and arguments as parsed JSON, not as text that the developer has to extract.

The guarantee comes from how the providers implement it. Three mechanisms appear, sometimes combined.

Constrained decoding restricts the model's token sampling to only those tokens that produce valid JSON matching the declared schema. The model literally cannot sample a token that would break the schema; the sampler enforces well-formedness during generation.

Grammar-aware sampling uses a finite-state machine or context-free grammar to gate token sampling. The model can only emit tokens that the grammar permits at the current state.

Fine-tuning trains the model to emit valid tool-call JSON consistently for tool-use scenarios, supplementing the sampling-time enforcement.

Whichever mechanism the provider uses, the developer-visible guarantee is the same: when the API returns a tool call, the JSON is well-formed. The runtime can parse it deterministically without regex, without try-except blocks for malformed JSON, and without fallbacks for ambiguous format.

Why this matters: parser failures vanish

Provider surface and the open-weight catch-up

The relationship to MCP

What function calling does not solve

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's tools parameter in chat completions returns tool_calls in the response; GPT-5.5 and o3 both support function calling natively.
Anthropic's Claude Opus 4.7 and Sonnet 4.6 support tool_use content blocks, the structured tool-call format that LangGraph and CrewAI build their agent loops around.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow is function calling different from MCP?

Function calling is inference-layer: how the model emits a tool call inside one inference request. MCP is system-layer: how the runtime advertises tool capabilities and routes calls to server processes. They are complementary; a 2026 agent uses both. Function calling makes the model's emission reliable; MCP makes the integration of external tools standardized.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing function calling with prompt-engineered tool calls. Prompt-engineered tool calls rely on the model writing parseable text; function calling guarantees structured JSON at the token-sampling layer.

60 second bullets to scan on the way to the call.

Define function calling as a provider-side feature emitting structured tool-call JSON.
Name the load-bearing property: structure guaranteed at the token-sampling layer.

Primary sources. Browse if you want the original framing.