Constrained decoding: making a model's output obey a schema
Grammar-guided generation compiles a schema into a token-level index so an output is guaranteed structurally valid by construction — a stronger guarantee than asking a model nicely to follow a format.
AI-assisted / research-based
This field note was drafted with AI assistance and synthesizes publicly available research papers and disclosed industry practice on an emerging AI technique. It is not based on confidential deployment data, is not investment, legal, medical, or security advice, and every primary claim links directly to its source so you can verify it yourself.
Asking a model to "respond only in valid JSON" is a request, not a guarantee. Downstream code that parses that response still has to handle the case where the model adds commentary, breaks the schema partway through, or produces output that is almost — but not quite — valid. Constrained decoding removes that uncertainty at its source, before a single invalid token can be generated.
A prompt instruction is advice the model can ignore. A token mask is a constraint it cannot.
Reformulating generation as transitions through a grammar
Efficient Guided Generation for Large Language Models ↗, by Willard and Louf, reformulates neural text generation in terms of transitions between the states of a finite-state machine. A regular expression or a context-free grammar defines which transitions are legal; the paper's method compiles that grammar into an index over the model's vocabulary, identifying — at each state the grammar can be in — exactly which tokens would keep the output on a legal path.
Invalid tokens are never reachable, not just discouraged
At each decoding step, the model's probability distribution over the next token is masked against this index: tokens that would violate the grammar are excluded from sampling entirely, not merely discouraged by a lower probability. The model only ever samples from whatever remains. Because the index is precomputed from the grammar rather than recomputed per token from scratch, the paper reports this approach adds little overhead to the generation process while significantly outperforming prior guided-generation approaches — and the reference implementation, the open-source library Outlines, is model-agnostic, working with any model exposing next-token probabilities.
The guarantee is structural, not semantic
It is worth being precise about what this technique actually guarantees. A JSON schema compiled into a token mask guarantees the output will parse as valid JSON matching that schema — it says nothing about whether the values inside that valid structure are correct. A schema-valid response can still contain a hallucinated field value, an incorrect entity ID, or a fabricated number, all inside perfectly well-formed JSON. Constrained decoding solves the class of failure where a downstream parser crashes on malformed output; it does not solve the class of failure where the content is wrong but the format is right.
Overly restrictive or poorly designed grammars carry their own risk: forcing generation down a narrow legal path can sometimes degrade the model's actual reasoning quality if the grammar doesn't leave room for the intermediate steps the model would otherwise use. The grammar itself deserves the same design scrutiny as a prompt.
Valid shape is not the same claim as correct content
What this changes in production
Tool-calling and function arguments
A malformed function argument doesn't just look wrong to a human reader — it breaks the downstream system call that depends on it. Constraining tool-call arguments to their declared schema at generation time removes an entire class of integration failure that a retry loop would otherwise have to handle.
Structured extraction pipelines
Pulling fields from documents into a fixed schema for storage or further automated processing benefits directly from a hard structural guarantee — a partially-parsed or malformed extraction is often worse than a clearly failed one, because it can silently corrupt whatever consumes it downstream.
API responses and agent-to-agent messages
Where a receiving system parses model output programmatically and cannot tolerate stray text around a structured payload, decoding-time constraints remove the need for a fragile "strip everything before the first curly brace" post-processing step.
A production checklist
Before relying on constrained decoding in a production path, the team should be able to answer:
- Does the constraint guarantee structural validity only, and is that distinction clear to whoever consumes the output — or is a valid schema being mistaken for a correct answer?
- Has the grammar itself been reviewed for whether it leaves the model enough room to reason before committing to structured output, or does it force premature commitment?
- What is the actual latency overhead of the constraint on this specific model and grammar, measured under production load rather than assumed from the paper's benchmarks?
- Is the constrained-decoding library being used compatible with the specific model and serving stack in production, including any custom sampling logic already in place?
- Does the pipeline still validate field values — types, ranges, referential integrity — on top of the guaranteed structural validity?
- What happens when the grammar itself has a bug or an edge case the schema didn't anticipate — is there a clear failure mode rather than a silent one?
Research referenced
- Willard and Louf (2023), Efficient Guided Generation for Large Language Models ↗.