JSON mode biases the sampler toward valid JSON; constrained decoding restricts the token space to a grammar. The harder the guarantee, the less the prompt has to spell out the shape.
Imagine filling out a form on paper. JSON mode is the difference between writing a polite sentence at the top, please use clear handwriting and keep names in the name box, and handing someone an actual form with boxes that physically prevent them from writing the answer outside the box. The sentence helps, but a person in a hurry can still write outside the lines. The form with real boxes makes that impossible. Constrained decoding is the printed form. JSON mode is the polite sentence. Once you have the printed form, you do not need to spend as many words asking for neat handwriting; the structure does the work.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
If you only need JSON out of an LLM and you do not look closely, JSON mode and constrained decoding feel like the same feature with two names. They both make structured output happen. They both involve a flag on the API call. They both produce something that parses.
Look closer and they sit on different layers of the stack. JSON mode is a learned bias plus light post-processing. Constrained decoding is a hard mask on the sampler's output distribution. The difference shows up exactly where it matters: on the long tail of inputs where the model would otherwise produce a near-miss shape.
From a context-engineering point of view, the distinction is also a question about who carries the schema. Either the prompt carries it (JSON mode) or the sampler carries it (constrained decoding). That choice changes how much of your token budget structure descriptions eat and how reliable your downstream parsing is.
What JSON mode actually does
JSON mode is a sampler-side bias. When you set response_format={'type': 'json_object'} (OpenAI's older mode) or equivalent on Anthropic or Gemini, the provider does some combination of (1) post-training the model to produce JSON when the flag is set and (2) light decode-time bias toward bracket-friendly tokens and away from prose openers.
The guarantee is at the syntactic level only. You get JSON that parses. You do not get JSON that matches any particular schema, because the flag does not know your schema. It cannot. You only told the API to produce JSON; you did not give it a contract.
What this means for the prompt
The prompt becomes the schema. You write something like 'Output a JSON object with fields user_id (string), score (number between 0 and 1), tags (array of strings, at most 5 items).' Those instructions are how the model learns the shape you want. Drop them and the model emits the shape it thinks is right, which is often almost but not quite your shape.
The failure mode is well-formed JSON that fails schema validation. Field names drift. Optional fields are sometimes included and sometimes not. Numbers come back as strings. Each of these is rare on a single call and certain across a population of calls. Production stacks that use JSON mode invariably wrap it in a retry on validation loop (Instructor, Pydantic AI) to cover the long tail.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Constrained decoding: schema is enforced sampler-side, prompt stays short
from openai import OpenAI
from pydantic import BaseModel
class Extraction(BaseModel):
user_id: str
score: float
tags: list[str]
client = OpenAI()
resp = client.responses.parse(
model="gpt-5.5",
input="Extract the user id, sentiment score, and tags from this review: ...",
response_format=Extraction, # strict JSON-Schema-constrained decoding
)
result: Extraction = resp.output_parsed
# No verbose 'output exactly this shape' instructions needed in the prompt.| Dimension | JSON mode | Constrained decoding |
|---|---|---|
| Mechanism | Output distribution biased toward JSON | Logits masked to enforce a grammar |
| Guarantee | Well-formed JSON | JSON that matches the supplied schema |
| Prompt burden | Must describe the schema in words | Schema lives outside the prompt |
| Failure mode | Wrong field names or types | Forced fills, off-distribution reasoning |
| Content quality | Higher on creative tasks | Can degrade under heavy constraint |
| Typical use | Loose extraction, creative with shape | Strict extraction, tool calls |
Real products, models, and research that use this idea.
- OpenAI's GPT-5.5 ships response_format=json_schema in strict mode, which is constrained decoding against a JSON Schema you supply.
- Anthropic Claude Opus 4.7 tool use enforces tool input schemas via constrained decoding under the hood.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you combine chain-of-thought reasoning with strict structured output?
Two-call pattern: first call generates free-text reasoning; second call (constrained-decoded) extracts the final structured answer from the reasoning. Or use a single call with a reasoning field allowed by the schema.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling them the same feature with different names. JSON mode is a soft bias; constrained decoding is a hard grammar restriction. The difference shows up at the long tail of inputs.
60 second bullets to scan on the way to the call.
Define JSON mode in one sentence
Define constrained decoding in one sentence
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.