**Grammar-constrained decoding** restricts the model's token choice at each step to tokens that keep the output syntactically valid. The correct answer is B; it wins in three specific regimes.
The contrast is between two approaches to structured output. The grammar approach (Outlines, llama.cpp grammars, some local inference servers) builds a state machine over the schema and masks any token at each decode step that would break it. The output is guaranteed valid. The cost is roughly 10 to 20% extra latency and a slight quality hit because the constraint can rule out reasoning paths the model would have taken.
The repair approach is simpler: let the model generate freely, parse the result, and if it fails, ask the model to fix it. About 95% of calls succeed first try; the other 5% pay a retry round trip.
**When grammar wins.** Zero tolerance environments (a downstream parser will crash, an embedded system has no error path, a regulator requires guaranteed schema compliance). High QPS systems where a 5% retry rate times millions of QPS adds up to more compute than the 15% per call grammar overhead. Regulated industries where 'we sometimes return malformed output' is not an acceptable answer.
**When repair wins.** Most other cases. Lower latency in the common path, full access to the model's reasoning, easy to mix with hosted APIs that do not expose token level grammars.
The other options are tempting traps. Option A overclaims; grammar is not strictly better. Option C is wrong; hosted Structured Outputs from OpenAI and Anthropic do offer schema enforcement but it is a different mechanism. Option D invents a depth threshold that does not exist.
**Grammar-constrained decoding** beats a repair loop in three named regimes: zero tolerance for malformed output, repair cost dominating at scale, and regulated environments where format compliance is mandatory. Answer B.
Grammar based decoders (Outlines, llama.cpp grammars, XGrammar) restrict the next-token distribution to tokens that keep the output valid against a schema or grammar. Output is guaranteed parseable. The cost is roughly 10 to 20% latency and a measurable quality hit on harder tasks because the constraint can prune reasoning paths.
Repair loops generate freely, validate, and retry on failure. They are cheaper in the common case because there is no decode time overhead and no reasoning restriction. The long tail can be expensive though. 5% retries at high QPS dominate the grammar latency cost. Any unrecoverable failure either crashes the downstream parser or requires fallback logic.
The three winning regimes for grammar are concrete. Embedded systems or downstream parsers that cannot recover from invalid input. High QPS production where retry economics flip. Regulated environments (finance, healthcare, gov) where 'sometimes invalid' is unacceptable.
Option A overstates the win; grammar is not strictly better. Option C confuses grammar based decoding with hosted Structured Outputs, which are related but different. Option D invents a depth threshold; repair loops handle deep nesting fine.
Quick reference
TL;DR
Grammar-constrained decoding wins over a repair loop when you cannot tolerate any malformed output, when retry cost dominates at high QPS, or when regulations require format guarantees.
Imagine you have to fill out a paper form perfectly. Option A: write whatever you want, then erase and redo mistakes (the repair loop). Option B: use a fancy pen that physically refuses to draw outside the form's lines (constrained decoding). Option B is always valid but slower and the strokes feel a bit stiff. Option A is fast and natural but sometimes you have to redo a section. If you submit forms once a week and the office accepts retries, Option A is fine. If a robot reads the form and crashes on any error, or if you fill out a thousand forms an hour and erasing is too slow, Option B starts to win.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks requiring rich chain of thought before a structured answer Grammar masks can prune reasoning tokens, hurting quality. Either leave a permissive thinking block first or use a two pass decode that reasons free then extracts.
Hosted only stacks where you cannot intercept logits Outlines and llama.cpp grammars need token level access. On managed APIs use the provider's Structured Outputs or tool use feature instead; same goal, different mechanism.
Low QPS apps with tolerant downstream consumers A repair loop is simpler, lower latency in the common case, and preserves full reasoning. The grammar overhead is not worth paying for occasional failures.
Quick scan · 2–4 min
Everything important, quickly.
4 min: grammar mask mechanism + latency and quality cost + three winning regimes + repair loop economics + hosted Structured Outputs as alternative + reasoning preservation tricks.
Real-world examples
Real products, models, and research that use this idea.
1Outlines and XGrammar are the standard open source token level grammar decoders, often used in front of llama.cpp or vLLM for local Llama 4 deployments.
2OpenAI Structured Outputs (GPT-5.5) and Anthropic tool use on Claude Opus 4.7 give schema guarantees server side; both are the hosted analog when you cannot run your own decoder.
3Regulated fintech workflows that emit a strict transaction schema use grammar decoders or hosted Structured Outputs because downstream ledger systems do not tolerate malformed input.
4High QPS extraction services (search indexing, log enrichment) reach for grammar decoding once retry cost on free generation exceeds the per call grammar overhead.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can grammar-constrained decoding hurt quality on reasoning tasks, and how do practitioners work around it?
Sign in or go Pro to see the approach for this follow-up.
QHow does OpenAI Structured Outputs differ mechanically from Outlines or XGrammar?
Sign in or go Pro to see the approach for this follow-up.
QAt what QPS does the math typically flip in favor of grammar over repair?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing grammar-constrained decoding is strictly better. It guarantees format but can slightly reduce reasoning quality and adds latency; the right choice depends on failure tolerance and QPS, not just elegance.