Compare BAML, Outlines, OpenAI Structured Outputs, and Pydantic with retry on what guarantee each actually gives you

Structured output is the problem of getting an LLM to return data that matches a specific schema, like a JSON object with named fields and types. Four common tools take very different approaches to the problem. **BAML** is an application-layer tool. You define a typed function in BAML's DSL, codegen produces a client in Python or TypeScript. The client calls the model, parses the response, and retries on parse failure. You get type-safety at the call site and eventual valid output with bounded retries. The failure mode is that a pathological prompt can exhaust the retry budget without ever parsing. **Outlines** is a decoding-layer tool. It derives a finite-state machine from your schema and masks the next-token logits during sampling so that invalid tokens have zero probability. Invalid output cannot be produced at all. The failure mode is that this only works on models where you control the sampling, usually self-hosted models, and the schema can degrade quality if it forces tokens the model would not naturally pick. **OpenAI Structured Outputs** is provider-side constrained sampling. Same kind of guarantee as Outlines but only available on OpenAI endpoints that opt in. The failure mode is vendor lock and the limited model list. **Pydantic + retry** validates the model's output after it is generated. If validation fails, you re-prompt with the error message. The guarantee is the weakest: you just get told if it is wrong, you do not prevent it.

The four approaches live at three different points in the LLM call lifecycle: before the first token (codegen), during sampling (decoding), and after the response (validation). Anchor each tool to a point and the comparison clicks. **BAML** operates at the application layer with codegen. You declare a typed function in `.baml` source files; the toolchain generates a typed client at build time. The client templates the prompt, calls the model, parses the response, and runs bounded retries on parse failure. Guarantee: type-safety at the call site plus eventual valid output within the retry budget. Failure mode: adversarial inputs exhaust the budget and surface as exceptions. **Outlines** operates at the decoding layer. It derives a finite-state machine from your schema and masks the next-token logits during sampling so invalid tokens get zero probability. Guarantee: invalid output cannot be produced. The strongest of the four. Failure modes: requires sampling control (mostly self-hosted models), can degrade quality if the schema forces low-probability tokens, FSM construction has overhead on complex schemas. **OpenAI Structured Outputs** is provider-side constrained sampling on supported models (`gpt-4o-2024-08-06` and later). Same guarantee as Outlines on the supported endpoints. Failure modes: vendor lock-in and a limited model list. **Pydantic + retry** is the weakest. Let the model generate freely, validate against a Pydantic model, re-prompt with the error on failure. Guarantee: detection only. Failure modes: silent partial validation (schema valid, values wrong) and unbounded retry cost when the model never converges. The four are not substitutes. They sit on a prevent versus detect axis; pick by provider, cost, and certainty needs.