Match each structured-output approach to the layer at which it enforces correctness

Match pairs·Hard·4.0 · 0·~2 min·Asked atOpenAI Anthropic Microsoft

Attempt it

Compile-time codegen of a typed client plus retry on parse failure

FSM-constrained token sampling, invalid tokens cannot be emitted

Constrained sampling on the provider side for supported models

Application-layer schema validation that re-prompts on failure

Generation in a constrained grammar via context free grammar inference rules

Structured output is the problem of getting an LLM to produce JSON (or any other format) that conforms to a schema. There are five distinct technical approaches in the 2026 landscape, and they live at three different layers of the stack. **Decoding-layer approaches (invalid output cannot exist):** - **Outlines** wraps the model's token-sampling step with a finite-state machine derived from the schema. At every step, only tokens that keep the output valid are eligible to be sampled. An invalid completion is not just unlikely; it is impossible. - **Llama.cpp and vLLM GBNF grammars** do the same idea with a context-free grammar: the grammar defines what tokens are legal at each parser state, and the sampler only emits legal tokens. - **OpenAI Structured Outputs** is the same idea moved to the provider side. You pass a JSON schema; the OpenAI server constrains its own sampling. Supported on `gpt-4o`, `gpt-5`, and `gpt-5.5` family. **Application-layer approach (invalid output is detected and recovered):** - **Pydantic + LangChain OutputFixingParser** lets the model generate freely, then validates against a Pydantic schema. On validation failure, it makes a second LLM call asking the model to fix the parse error. Recovery, not prevention. **Compile-time approach (a typed client + retry):** - **BAML** generates a typed client from a schema at build time. The generated client wraps the LLM call, parses the response into the typed schema, and retries on failure with a structured error message back to the model.

**The landscape sorts on one axis: at what layer is correctness enforced?** **Decoding-layer enforcement (invalid output is impossible).** - **Outlines.** Compiles the schema into a finite-state machine. At every decoding step, the FSM masks logits so only tokens that keep the output valid can be sampled. Guaranteed valid by construction. - **Llama.cpp / vLLM GBNF grammars.** Same idea with a more expressive substrate: a context-free grammar. Lets you constrain to any structured format (JSON, SQL, custom DSLs). - **OpenAI Structured Outputs.** Provider-side constrained sampling. Pass a JSON schema; OpenAI's server enforces it. Available on `gpt-4o`, `gpt-5`, and `gpt-5.5`. Anthropic tool-calling achieves the same via a different mechanism. **Application-layer enforcement (invalid output is detected and recovered).** - **Pydantic + LangChain OutputFixingParser.** Generate freely, validate against a Pydantic schema, on failure call the model again with the error to fix. Bounded by the LLM's self-correction ability; usually succeeds in one retry. **Compile-time enforcement (typed client + retry).** - **BAML.** Generates a typed client from a `.baml` schema at build time. The client wraps the call, parse, and retry policy; the application code calls a typed function and gets a typed object. **The architectural insight.** Decoding-layer is the strongest guarantee but requires control of the decoder (self-hosted or provider-side support). Application-layer is provider-agnostic but only recovers, does not prevent. Compile-time wraps the call in a typed client; underlying enforcement is still application-layer retry.

Quick reference

TL;DR

Structured-output tools live at three layers: decoding (Outlines, GBNF, provider-side), application (Pydantic + retry), and compile-time (BAML). The layer determines what guarantees you get.

Memory aid

Easy to grasp

Imagine three ways to stop a typo in an essay. The first is a magical pen that physically refuses to write any letter that would create a typo (decoding-layer). The second is a kind editor who reads every sentence and asks you to rewrite it if a typo slipped through (application-layer). The third is a typewriter that was built with your specific essay's vocabulary hard-coded into its keys, so only valid words can be typed (compile-time codegen). All three prevent typos in the final essay; they just do it at very different moments.

Key concepts

LLM frameworks

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 min: three-layer taxonomy + decoding-layer mechanisms (FSM, GBNF, provider-side) + application-layer retry + compile-time codegen + production stack combining layers + when constrained decoding hurts.

Real-world examples

Real products, models, and research that use this idea.

OpenAI Structured Outputs first shipped in 2024 and by 2026 is the default for production extraction workloads on gpt-5.5 and the o5 reasoning family.
BAML adoption in 2025-2026 grew in TypeScript-heavy teams that wanted typed clients and compile-time schema checking.
Outlines is the standard pick for self-hosted Llama 4 Maverick and DeepSeek V4 deployments where decoding-layer constraints are needed.
Llama.cpp's GBNF grammars are widely used in local-inference pipelines (LM Studio, Ollama-adjacent tooling) for structured output on consumer hardware.
Anthropic's Claude Opus 4.7 supports structured output through its tool-calling shape; for Claude users this is the closest analogue to OpenAI Structured Outputs.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through what an FSM-constrained decoding step actually does mechanically.

QWhy does OpenAI's `response_format={'type': 'json_object'}` not give the same guarantee as Structured Outputs?

QHow does BAML differ from just using Pydantic schemas with LangChain?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating Outlines, BAML, and OpenAI Structured Outputs as substitutes that solve the same problem at the same layer. They are stacked layers, not alternatives; a production stack often uses more than one.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.