Grammar-constrained decoding support across LLM APIs

Grammar-constrained decoding LLM API support determines whether you can force a model to emit output matching a formal grammar—JSON Schema, regex, or context-free grammar—rather than hoping prompt engineering holds. This post compares how the major hosted APIs expose that control, where they fall short, and what it costs you in latency and dollars.

The contenders

We examine four hosted options that engineers actually call from production:

• OpenAI (chat completions with response_format strict JSON Schema)
• Anthropic Claude (tool-use schemas with forced tool_choice)
• Together AI (native grammar parameter backed by Outlines)
• Groq (JSON mode on LPU-hosted open weights, no arbitrary grammar)

A gateway such as n4n.ai sits in front of these and exposes one OpenAI-compatible endpoint that fronts 240+ models, letting you switch between OpenAI’s strict schema and Together’s regex without rewriting clients.

Capabilities

OpenAI’s structured outputs use constrained decoding to guarantee the response matches a supplied JSON Schema when strict: true is set. It supports nested objects, arrays, enums, and additionalProperties: false. It does not support arbitrary context-free grammars or regex.

resp = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {"name": {"type": "string"}},
                "required": ["name"],
                "additionalProperties": False
            }
        }
    }
)

Anthropic has no response_format endpoint. You declare a tool with an input_schema (JSON Schema subset) and force the model to call it via tool_choice. The output is the tool input, not a raw grammar match. No regex or CFG.

resp = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=512,
    tools=[{"name": "extract",
            "input_schema": {"type": "object",
                             "properties": {"name": {"type": "string"}},
                             "required": ["name"]}}],
    tool_choice={"type": "tool", "name": "extract"}
)

Together AI exposes a grammar field on its chat completion endpoint. It accepts {"type": "regex", "value": "..."} or {"type": "json", "schema": {...}} and uses Outlines for server-side constrained generation. This is the only hosted API here that will take a raw regex or CFG.

resp = client.chat.completions.create(
    model="meta-llama/Llama-3-70b-chat-hf",
    messages=[{"role": "user", "content": "Name a cat."}],
    grammar={"type": "regex", "value": r'\{"name": "[A-Za-z]+"\}')
)

Groq mirrors OpenAI’s json_object mode but does not guarantee schema adherence—only that the output is valid JSON. There is no grammar parameter and no strict mode.

Price and cost model

OpenAI bills per token regardless of constraint; strict mode may add a small prefill overhead but no separate line item. Anthropic bills input/output tokens; forced tool use does not change pricing. Together AI charges per token with no grammar surcharge, but smaller open-weight models are typically 3–5× cheaper per 1K tokens than GPT-4o. Groq prices LPU inference aggressively per token, often the cheapest of the four for equivalent parameter counts, but you pay for a weaker guarantee.

None of the providers meter the constraint compilation step separately. If you self-host via vLLM or TGI, the cost is GPU time, not token price.

Latency and throughput

Constrained decoding adds a mask-computation step at each token. On OpenAI’s infrastructure this is hidden behind the API; p50 latency for a 100-token strict JSON response is comparable to unconstrained. Together’s Outlines path adds measurable overhead on smaller GPUs—expect 10–20% slower decode versus unconstrained Llama-3-70B. Groq’s LPU is fastest raw tokens/sec, but because it lacks true constraints, you may spend round-trips validating and repairing output, negating the win. Anthropic’s tool-force path is on par with normal Claude latency.

Ergonomics

OpenAI’s SDK accepts response_format natively; the schema must be JSON Schema draft 2020-12 with restrictions. Anthropic requires you to wrap extraction in a tool and parse tool_use blocks—verbose but workable. Together’s grammar is a single extra parameter but the Python SDK does not type-check it; you ship dicts. Groq is a drop-in OpenAI client swap with response_format={"type":"json_object"}.

For regex needs, only Together gives you a first-class escape hatch. If you need CFG, you are self-hosting or using Together.

Ecosystem and tooling

OpenAI’s strict mode is supported by LangChain, Pydantic, and OpenAI’s own pydantic beta. Anthropic tool use is universal in agent frameworks. Together’s grammar is less integrated—you often write raw dicts or use Outlines directly. Groq inherits the OpenAI ecosystem but loses strictness.

When you route through a single endpoint that fronts multiple providers, you trade vendor-specific ergonomics for portability. The gateway forwards your response_format or grammar to the backend that understands it.

Limits and caveats

• OpenAI strict mode forbids additionalProperties, nullable unions beyond basics, and recursive schemas.
• Anthropic tool schemas reject default, $ref, and formats like date-time.
• Together’s regex must be PCRE-compatible; complex CFGs can blow up the mask cache.
• Groq JSON mode can still emit trailing prose after the JSON; you must trim.
• All constrained paths disable sampling freedom: temperature is effectively ignored for masked positions.

Comparison table

Which to choose

You need guaranteed JSON matching a precise schema and are already on OpenAI: Use response_format strict mode. It is the lowest-friction production path and frameworks already speak it.

You live in Claude tool-calling loops: Force tool_choice and treat the tool input as your structured output. Do not expect regex.

You need regex or a custom context-free grammar: Together AI is the only hosted API here that supports it natively. Accept the slight latency tax and write your grammar carefully.

You want cheapest tokens and can validate client-side: Groq with json_object mode plus a Pydantic retry loop wins on speed and price, but you own the repair logic.

You run multi-provider and hate vendor lock: Route through a single OpenAI-compatible gateway that fronts these backends, set the constraint field per route, and keep your client code stable while you shift models based on cost or degradation.