Describe why structured-output mode quietly raises the per-call token bill

Attempt it

Switching from free-text completions to strict-JSON or structured-output mode usually raises the realized cost per useful answer by 10-40%, even when the underlying information returned is identical. Name the three mechanisms responsible and explain how each adds tokens or wall-clock time.

Free · 2 AI evals / day

**Three independent forces push the realized cost up.** None of them is about the actual information the model returns. Two add tokens; one adds wall-clock time. First, **constrained decoding**. JSON mode forces the next-token distribution through a grammar or finite-state mask (Outlines, JSONSchema-FSM, llguidance). At every step the mask zeros out tokens that would violate the schema. That mask interacts badly with speculative decoding: candidate tokens from the draft model that violate the schema get rejected by the mask, so the verifier accepts fewer tokens per round. The model still runs at the same per-step bandwidth, but the per-token wall-clock effectively rises because speculation is less efficient. Second, **envelope tokens**. The JSON syntax itself is real billed output. Keys are repeated verbatim in every record (`"customer_name": ..., "order_id": ...`). Quotes, braces, brackets, commas, and escape sequences all count as output tokens. A free-text answer that would have been 80 tokens easily becomes 130 tokens when wrapped in JSON, a 60% uplift just from envelope. Third, **parse-failure retries**. The model occasionally emits a near-miss: an unclosed brace, an invalid escape, a value that fails schema validation. The application retries, paying for prompt input and full output a second time. Even a 2-3% failure rate adds up across a high-volume route. Combined, these typically land at a 10-40% net output-token bill increase over the free-text equivalent, depending on response shape. The fix is not to abandon structure; it is to use lean schemas, prefer providers with grammar-aware speculation, and budget for retries.

**Three compounding mechanisms.** None of them is about the underlying information; all of them add to the realized cost. First, **constrained decoding lowers speculation acceptance**. JSON mode runs every sampling step through a schema-aware grammar mask. The mask zeros out tokens that would violate the schema, so the next-token distribution narrows. Speculative decoding, which depends on the draft model producing candidates that match the target's distribution, takes a hit: candidates that the draft predicted but the mask forbids get rejected by the verifier. Acceptance rate drops, draft rounds yield fewer accepted tokens, and wall-clock latency per token rises even though output length is unchanged. Second, **the JSON envelope is real billed output**. Keys are repeated verbatim every record (`customer_name`, `order_id`, `address`). Quotes, braces, brackets, commas, and escape sequences are tokens. A response that would be 80 prose tokens easily becomes 130 JSON tokens. The uplift is biggest on short structured replies where the envelope dominates the payload, and smallest on long structured replies where the value content amortizes the envelope cost. Third, **parse failures trigger retries**. Even a well-tuned structured-output mode emits the occasional malformed payload: an unclosed string, an invalid escape, a value that violates schema validation. The application retries, paying for the input prompt and the output a second time. At a 2-3% failure rate, that adds 2-3% to the realized cost on top of the other axes. The combined effect lands at 10-40% above the free-text equivalent in practice. The fix is to use the leanest schema that works (shorter keys, fewer redundant fields), prefer providers with grammar-aware speculative decoding (some stacks adapt the draft model to respect the schema), cache the compiled grammar so the FSM is reused across requests, and budget explicitly for a small retry tail.

Quick reference

TL;DR

Structured-output mode adds 10-40% to output cost via three axes: constrained-decoding mask hurts speculation acceptance, JSON syntax adds envelope tokens, and parse failures cause retries.

Memory aid

Easy to grasp

Imagine asking someone for an answer in two ways. First, just tell me. Second, fill out this fixed form with labeled boxes (name, age, address). The second response always takes more words because of the labels, the brackets, and the formal structure, even when the underlying information is the same. On top of that, the person has to slow down while writing to make sure every label is spelled right and every bracket matches; rules force corrections. And occasionally they hand back a form that has a typo and you ask them to redo it, doubling the work for that one. That is what JSON mode does to an LLM: more words for the envelope, slower per word because of the rules, and the occasional do-over.

Key concepts

Structured output Constrained decoding Speculative decoding

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: three cost axes + constrained sampling and speculation drop + envelope overhead + retry tax + schema design tips + grammar-aware speculation.

Real-world examples

Real products, models, and research that use this idea.

Outlines and llguidance are the production-grade grammar-aware sampling libraries used by vLLM, TGI, and Together AI to enforce JSON Schema during decoding.
OpenAI Structured Outputs (introduced 2024) uses a constrained sampler and documents that strict schema mode can be slightly slower per request than free-text completion.
Anthropic Claude Opus 4.7 and Sonnet 4.6 support tool use with JSON Schema; their documentation recommends compact field names to minimize the envelope token tax.
Together AI published benchmarks in 2026 showing speculative-decoding acceptance dropping from 0.72 to 0.48 under JSON Schema constraints on Llama 4 Maverick.
Production teams at Stripe and Notion have documented 20-30% output token cost increases when migrating routes from free-text to structured outputs, citing all three axes.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does grammar-aware speculative decoding restore some of the lost acceptance rate?

QWhat is the right schema-design philosophy for low cost under structured output?

QHow do you measure the realized cost of structured output mode separate from the model itself?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing JSON mode is free because the model already produces structured-looking text. Constrained decoding actively masks tokens, lowering speculative-decoding acceptance, and the schema envelope tokens are real billed output.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.