What is JSON mode (response_format) in LLM APIs and what does it guarantee?
JSON mode is a provider flag that forces valid JSON output, usually by constrained decoding. Schema modes (Structured Outputs, strict tool use) also enforce your declared field names and types.
Imagine the model is filling in a form, and the form has rules: this field must be a number, that field must be a list, the whole form must end with a closing brace. Without JSON mode, the model tries to follow the rules but might accidentally leave a comma in the wrong place or wrap the form in friendly chitchat. With JSON mode on, the form has rails. At every step the form-filler is only allowed to write characters that keep the form valid. Invalid characters are not even on the menu. The form comes back parseable, unless the model runs out of space and stops halfway. The model might still get a value wrong (writing the wrong number for a field), but you do not need to wrestle with broken JSON anymore.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
JSON mode is one of the most operationally important features that landed in LLM APIs between 2023 and 2025. Before it existed, every team using LLMs to produce structured data had to wrap the API call in retry on parse error logic. They had to deal with prose contamination. They also had to accept that some percentage of calls would fail to parse, no matter how carefully the prompt was written. JSON mode replaced all of that with a hard guarantee: the output parses, as long as generation is not cut off.
The usual technique behind the guarantee is constrained decoding at the token-sampling layer. The inference engine knows what valid JSON looks like (or what valid output under your declared JSON Schema looks like). At every generation step it masks out any token that would break validity. The model still does the heavy lifting of choosing values; the engine just makes sure those choices land inside the legal structural envelope.
Why JSON mode exists at all
The 2022-2023 era of structured output from LLMs was, in retrospect, a workaround. The pattern looked like this. Write a prompt that says 'respond with JSON in the following shape,' and show two or three few-shot examples of the desired JSON. Then parse the response, and retry on failure. Production code wrapped every call in a try-catch over JSON.parse and either fell back to a different prompt or returned an error.
The failure rate was not enormous but it was not zero. Models occasionally wrapped the JSON in 'Here is the response: ...' prose. They added trailing commas where the spec forbade them. They closed objects without all the expected fields. They hallucinated comments. Each failure mode had its own workaround (regex extraction, lenient parsers, repair prompts). The resulting code path was a tangle that every team doing structured extraction depended on.
JSON mode treats this as a sampling problem, not a model problem. The model has plenty of training data showing what valid JSON looks like and produces valid JSON most of the time. The failures come from the unconstrained sampling process occasionally drifting outside the valid structural envelope. If you constrain the sampling so it cannot drift outside that envelope, the failures vanish at the source. JSON mode does exactly this.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's Structured Outputs (response_format with json_schema and strict: true) guarantees output that matches your supplied JSON Schema, which SDKs can build from a Pydantic model.
- Anthropic's structured outputs (a JSON Schema in output_config.format) and strict tool use (strict: true on a tool) guarantee that Claude's response or tool_use input matches the schema. Both use grammar-constrained decoding.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does constrained decoding actually work at the token level?
Maintain a parser state during generation. At each step, build a mask over the vocabulary marking which tokens keep the output valid. Apply the mask to logits before softmax. The model never samples an illegal token.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming JSON mode also guarantees the values are correct. It guarantees the output parses and, in schema mode, conforms to the schema. Whether the model wrote the right number for revenue is a separate, prompt-level concern.
60 second bullets to scan on the way to the call.
Define JSON mode and name the underlying mechanism (constrained decoding)
Explain how the token-sampling mask is applied at each generation step
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.