Grammar-constrained decoding is an inference-time technique that restricts a language model’s token sampling to sequences that satisfy a specified formal grammar, such as JSON or a SQL subset. Unlike post-hoc validation, it guarantees every emitted token adheres to the schema, eliminating malformed output at the source. For AI agents that must call tools or emit structured state, grammar constrained decoding turns probabilistic text generation into a typed interface.
What grammar constrained decoding actually is
At its core, grammar constrained decoding interposes a deterministic filter between the model’s output logits and the sampling step. The model still produces a probability distribution over its vocabulary at each position. Before you draw a token, you consult a parser that knows the grammar and the partial string generated so far. Any token that would lead to a dead end—a sequence that cannot be completed to a valid sentence—gets assigned negative infinity logit. You then sample from the masked distribution.
This is not prompt engineering. You are not begging the model to “please output valid JSON.” You are mathematically forbidding it from doing anything else. The constraint is enforced by the decoding loop, not by the weights.
The grammar can be expressed in many forms: a JSON Schema, an EBNF file, a regular expression, or a Pydantic model. Under the hood these are compiled to a state machine that tracks the allowed continuations.
How it works under the hood
Token masking with a live parser
Suppose the grammar requires the output to start with {. After the model emits the BOS token, the parser sees an empty prefix. It knows the only valid next character class is {. The vocabulary contains thousands of tokens, but only those whose first byte is { (and which can be extended to a full brace) survive the mask. Everything else is zeroed.
As generation proceeds, the parser maintains a stack or DFA state. If the schema says we are inside a string value for the query field, the mask permits arbitrary Unicode tokens that don’t contain a closing quote, plus the quote-close token. It does not permit a stray comma or a new key until the string ends.
From grammar to finite-state automaton
Most practical implementations do not run a full LL(1) parser per token because that is slow. Libraries like Outlines convert the grammar to a constrained regular approximation or a specialized context-free parser with incremental execution. llama.cpp accepts GBNF (GGML BNF) and compiles it to a parse table before generation starts.
The key insight: the set of valid next tokens is a regular language at any fixed prefix depth for many useful grammars, especially JSON. That lets the engine precompute token prefixes and apply a fast bitwise mask over the logits tensor on the GPU.
Where the mask lives
The masking must happen inside the inference server, because it needs the raw logits. If you call a hosted model through a plain HTTP API that only returns completed text, you cannot enforce grammar after the fact. You need either a provider that supports constrained decoding natively, or a local/sidecar server (e.g., llama.cpp, vLLM with guidance, TRT-LLM) that exposes it.
Why agents need it
Deterministic tool calls
An agent that calls search(query) cannot afford to receive "query": "red shoes with a missing closing quote. A JSON parser downstream will throw, the agent loop catches the exception, retries, burns tokens, and maybe fails again. Grammar constrained decoding makes the first attempt structurally valid by construction.
Avoiding retry storms
Without constraints, even a 1% malformation rate compounds across a multi-step plan. If an agent makes 20 structured calls per task, the chance of at least one bad output is ~18%. With constraints, it is zero. That directly cuts latency and cost.
Compositional schemas
Agents often need to emit a wrapper object containing a thought string and an action discriminated union. Hand-written prompts fail on nested arrays. A formal grammar handles nesting naturally.
Concrete example: locked JSON for a support agent
We want the model to return exactly one of three actions plus a free-text query. Using Outlines:
import outlines
model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")
schema = {
"type": "object",
"properties": {
"action": {"type": "string", "enum": ["search", "buy", "exit"]},
"query": {"type": "string"}
},
"required": ["action", "query"]
}
generator = outlines.generate.json(model, schema)
result = generator("User: find red shoes under 50 dollars. Agent:")
print(result)result is a Python dict, not a string. The enum constraint means the model can only emit one of the three exact action strings; the query field accepts any string that respects JSON escaping. No json.loads try/except required.
If you run llama.cpp directly, the equivalent grammar file looks like:
root ::= "{" space "\"action\"" space ":" space ("\"search\"" | "\"buy\"" | "\"exit\"") space "," space "\"query\"" space ":" space string space "}"
string ::= "\"" ([^"] | "\\\"")* "\""
space ::= [ \n\t]*You pass it with --grammar-file response.gbnf. The engine never produces a token that violates that rule.
Common misconceptions
“JSON mode is the same thing”
OpenAI’s response_format: json_object only strongly biases the model toward emitting a JSON object. It does not guarantee the keys match your schema, nor that nested arrays are well formed. Grammar constrained decoding enforces the exact shape. JSON mode is a prompt-level nudge; grammar decoding is a hard constraint.
“It kills model creativity”
The constraint applies only to syntax, not semantics. Inside a string field the model can write poetry. Inside an enum it cannot invent a fourth action—which is exactly what you want for a dispatcher.
“The overhead is unacceptable”
Compiling the grammar is a one-time cost per request (or cached across requests with identical schema). The per-token mask is a tensor operation that typically adds single-digit milliseconds on modern GPUs. The saved retry cost usually dwarfs it.
“It works transparently on any provider”
False. If the inference endpoint does not expose logit masking, you cannot enforce grammar remotely. You must self-host or use a gateway that fronts models with native support. Some OpenAI-compatible endpoints merely pass through to providers that ignore the grammar parameter.
Running it in production
In an agent stack you usually separate the orchestration layer from the model layer. If you front models with an OpenAI-compatible endpoint such as n4n.ai, the constraint execution stays in the model server or a local proxy; the gateway routes and meters tokens but does not alter logits. Its per-token usage metering still lets you attribute cost to each constrained generation, and honoring client routing directives means you can pin a request to a model that supports native grammar compilation rather than one that silently drops the parameter.
For high-throughput agents, cache the compiled grammar. JSON Schemas repeat across calls; rebuild the automaton once per process. Stream the output token-by-token to the agent loop so partial structures can be inspected, but do not attempt to parse until the grammar signals completion.
Grammar constrained decoding is the difference between hoping your agent’s output parses and knowing it will. Ship the constraint, delete the validation retry loop, and move on to harder problems.