Contrast JSON mode and constrained decoding from a context-engineering point of view

Attempt it

JSON mode is a model-side switch that biases the output distribution toward valid JSON; the prompt still has to describe the shape in plain language and the model can still emit a shape that does not match the consumer's schema. Constrained decoding (grammars, regex, JSON-Schema-constrained sampling) hard-restricts the token space so the output literally cannot leave the grammar; the prompt can be shorter because the shape is enforced by the sampler, not begged for in words. Practical context implication: with constrained decoding you can drop the verbose 'output exactly this shape' instructions and reclaim those tokens; with plain JSON mode you must keep them or accept that some outputs will fail schema validation.

Your answer

Quick reference

TL;DR

JSON mode biases the sampler toward valid JSON; constrained decoding restricts the token space to a grammar. The harder the guarantee, the less the prompt has to spell out the shape.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine filling out a form on paper. JSON mode is the difference between writing a polite sentence at the top, please use clear handwriting and keep names in the name box, and handing someone an actual form with boxes that physically prevent them from writing the answer outside the box. The sentence helps, but a person in a hurry can still write outside the lines. The form with real boxes makes that impossible. Constrained decoding is the printed form. JSON mode is the polite sentence. Once you have the printed form, you do not need to spend as many words asking for neat handwriting; the structure does the work.

Key concepts

Context engineering Prompt engineering LLM frameworks

Quick scan · 2–4 min

Everything important, quickly.

5 minutes: define both features, name the sampler-bias vs sampler-restriction difference, walk the prompt-token reclaim, list one quality risk, name two 2026 implementations.

Code example

python

# Constrained decoding: schema is enforced sampler-side, prompt stays short
from openai import OpenAI
from pydantic import BaseModel

class Extraction(BaseModel):
    user_id: str
    score: float
    tags: list[str]

client = OpenAI()
resp = client.responses.parse(
    model="gpt-5.5",
    input="Extract the user id, sentiment score, and tags from this review: ...",
    response_format=Extraction,  # strict JSON-Schema-constrained decoding
)
result: Extraction = resp.output_parsed
# No verbose 'output exactly this shape' instructions needed in the prompt.

Side-by-side

Dimension	JSON mode	Constrained decoding
Mechanism	Output distribution biased toward JSON	Logits masked to enforce a grammar
Guarantee	Well-formed JSON	JSON that matches the supplied schema
Prompt burden	Must describe the schema in words	Schema lives outside the prompt
Failure mode	Wrong field names or types	Forced fills, off-distribution reasoning
Content quality	Higher on creative tasks	Can degrade under heavy constraint
Typical use	Loose extraction, creative with shape	Strict extraction, tool calls

Real-world examples

Real products, models, and research that use this idea.

OpenAI's GPT-5.5 ships response_format=json_schema in strict mode, which is constrained decoding against a JSON Schema you supply.
Anthropic Claude Opus 4.7 tool use enforces tool input schemas via constrained decoding under the hood.
Outlines (dottxt-ai/outlines) is the dominant open-source constrained-decoding library; works with vLLM, llama.cpp, and Transformers backends.
lm format enforcer in vLLM is a 2026 standard for self-hosted Llama 4 and Qwen 3.5 deployments.
Gemini 3.1 Pro's structured output mode accepts a responseSchema and constrains decoding to it.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you combine chain-of-thought reasoning with strict structured output?

Sign in or go Pro to see the approach for this follow-up.

QWhat goes wrong if you constrain decoding too aggressively on a small model?

Sign in or go Pro to see the approach for this follow-up.

QWhy do tool calls implicitly use constrained decoding?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.