Structured outputs and JSON mode across LLM providers

Building AI agents that call tools or emit parseable data demands a reliable structured outputs JSON mode LLM API. The big providers have each shipped their own flavor of constrained decoding or schema enforcement, but the semantics, limits, and ergonomics differ enough to bite you in production. This head-to-head compares OpenAI, Anthropic, Google Gemini, and Mistral on the dimensions that matter when you wire these models into a system.

The providers we’re comparing

We focus on the four hosted APIs that engineers actually hit from Python today:

• OpenAI – chat.completions with response_format
• Anthropic – messages with tool definitions
• Google Gemini – generativeModel.generate_content with response_schema
• Mistral – chat.complete with response_format

All four expose HTTP endpoints, but only one of them treats structured generation as a first-class decode constraint rather than a prompt convention.

Capabilities: how each enforces structure

OpenAI

OpenAI gives you two distinct modes. Loose JSON mode ("type":"json_object") only guarantees valid JSON, not shape. Structured Outputs ("type":"json_schema") uses a constrained decoder that rejects any token violating the schema. It supports a subset of JSON Schema draft 2020-12: objects, arrays, enums, anyOf, nullable, and $defs for recursion. It does not support pattern regex or arbitrary length bounds.

from openai import OpenAI
client = OpenAI()
resp = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[{"role":"user","content":"Extract name and age"}],
    response_format={
        "type":"json_schema",
        "json_schema":{
            "name":"person",
            "strict":True,
            "schema":{
                "type":"object",
                "properties":{
                    "name":{"type":"string"},
                    "age":{"type":"integer"}
                },
                "required":["name","age"],
                "additionalProperties":False
            }
        }
    }
)
print(resp.choices[0].message.content)

Anthropic

Anthropic has no response_format parameter. Structure comes from the tool-use protocol: you define a tool with an input_schema (JSON Schema draft 7) and force tool_choice. The model returns a tool_use block whose input is guaranteed to match the schema. This couples structure to the agent loop but is rock solid.

import anthropic
client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    tools=[{
        "name":"extract",
        "description":"Return person",
        "input_schema":{
            "type":"object",
            "properties":{"name":{"type":"string"},"age":{"type":"integer"}},
            "required":["name","age"]
        }
    }],
    tool_choice={"type":"tool","name":"extract"},
    messages=[{"role":"user","content":"Extract name and age"}]
)
print(resp.content[0].input)

Google Gemini

Gemini accepts response_mime_type="application/json" plus a response_schema (OpenAPI subset). The decoder constrains generation and strips unknown keys server-side. It handles deep nesting and enums well, but cannot enforce additionalProperties:false at decode time—it filters instead of rejecting.

import google.generativeai as genai
genai.configure(api_key="...")
model = genai.GenerativeModel("gemini-1.5-pro")
resp = model.generate_content(
    "Extract name and age",
    generation_config={
        "response_mime_type":"application/json",
        "response_schema":{
            "type":"object",
            "properties":{"name":{"type":"string"},"age":{"type":"integer"}},
            "required":["name","age"]
        }
    }
)
print(resp.text)

Mistral

Mistral mirrors OpenAI’s loose mode: pass response_format={"type":"json_object"}. No schema, no constraints. You must validate and retry on failure.

from mistralai import Mistral
client = Mistral(api_key="...")
resp = client.chat.complete(
    model="mistral-large-latest",
    messages=[{"role":"user","content":"Extract name and age as JSON"}],
    response_format={"type":"json_object"}
)
print(resp.choices[0].message.content)

Price and cost model

All four charge per output token at the same rate as unstructured generation. OpenAI Structured Outputs and Gemini constrained decode incur no explicit premium. Anthropic’s tool-use path consumes tokens for the tool definition in the context window but not extra per-call fees. Mistral’s JSON mode is free of surcharge. The only hidden cost is schema complexity: deeper nesting can raise prompt tokens indirectly via repeated schema echoes in some implementations.

Gateways such as n4n.ai provide per-token usage metering across these providers so you can attribute cost precisely without building four billing integrations.

Latency and throughput

Constrained decoding adds minimal overhead on OpenAI and Gemini; both use optimized parsers that run in parallel with token generation. Anthropic’s tool-use round trip is comparable to a normal completion. Mistral’s post-hoc JSON mode does not constrain decoding, so latency is identical to base model but you pay a retry cost if the model emits invalid JSON. Across providers, streaming works with structure except Mistral’s json_object (no token stream guarantee). In practice, time-to-first-token grows by single-digit milliseconds under strict schemas on OpenAI; Gemini’s filtering adds negligible post-processing.

Ergonomics

OpenAI’s Structured Outputs ships SDK support that validates the schema client-side and parses directly into pydantic. Anthropic’s tool schema is verbose but integrates with agent loops naturally. Gemini’s response_schema is clean but the Python SDK returns a string, not a parsed object. Mistral requires you to json.loads and handle malformed output. For quick prototyping, OpenAI wins; for agent-native design, Anthropic feels closest to function-calling idioms.

# OpenAI + pydantic (official helper)
from pydantic import BaseModel
class Person(BaseModel):
    name: str
    age: int
# client.beta.chat.completions.parse(...) returns parsed object

Ecosystem and limits

OpenAI: deepest ecosystem, pydantic plugin, largest model selection. Limit: strict mode rejects schemas with arbitrary recursion unless via $defs, and omits regex patterns. Anthropic: no standalone JSON mode, max tool schema size ~32KB, but supports most draft-7 keywords. Gemini: supports nested schemas and enums, but responseSchema can’t express additionalProperties:false strictly—it filters unknown keys server-side. Mistral: only json_object, no strict, max output 4k tokens typical.

Comparison table

Which to choose

Strictest schema compliance: Use OpenAI Structured Outputs when you need guaranteed parseable objects with zero post-validation. The strict flag removes a class of production incidents.

Agent tool-use loops: Anthropic’s tool_choice forcing is the most natural fit if your system already treats every model turn as a tool call. You avoid a separate JSON path.

Multimodal + structured: Gemini’s responseSchema works on image+text inputs, so if you extract structured data from photos, it’s the only option here that mixes modalities without separate OCR.

EU hosting / open weights: Mistral gives you json_object on European infrastructure and self-hostable models, but build your own validator.

Multi-provider resilience: If you want to write the structured outputs JSON mode LLM API call once and not rewrite per provider, an inference gateway like n4n.ai fronts 240+ models behind one OpenAI-compatible endpoint with automatic fallback when a provider is degraded, forwarding your schema hints without a translation layer. That removes the comparison’s operational tax.

Pick based on where the pain is: schema rigidity, agent idiom, modality, or jurisdiction. The APIs are converging, but the edges still matter.