JSON mode across providers: OpenAI, Anthropic, Gemini

JSON mode OpenAI Anthropic Gemini comparison matters because every provider implements structured output differently, and the differences leak into your retry logic, schema validation, and cost model. OpenAI pioneered the feature with response_format: { type: "json_object" }, Anthropic added tool-use coercion, and Gemini ships native schema enforcement via generationConfig.responseMimeType. None of them behave identically under the hood. This post breaks down the practical differences you hit in production.

What each provider actually ships

OpenAI’s JSON mode guarantees valid JSON syntax but not schema conformance. You get a json_object response type that forces the model to emit parseable JSON, yet the content can still omit required fields, hallucinate keys, or violate type constraints. The workaround is function calling with strict: true, which enforces a JSON Schema at inference time — but only on gpt-4o, gpt-4o-mini, and newer models.

Anthropic doesn’t have a dedicated JSON mode. Instead, you use tool use with a single function whose input schema matches your desired output. The model emits a tool call block; you extract the input field as your JSON. This works on all Claude 3 models (Opus, Sonnet, Haiku) and 3.5 Sonnet. The model can still refuse to call the tool, so you need fallback handling.

Gemini 1.5 Pro and Flash support responseMimeType: "application/json" with an optional responseSchema (a subset of JSON Schema). When you provide a schema, the decoder constrains token generation to valid structures — closer to OpenAI’s strict mode than to its basic JSON mode. Gemini 1.0 models lack this entirely.

Schema enforcement: strict vs. best-effort

OpenAI’s strict: true in function calling compiles your JSON Schema into a deterministic finite automaton that guides token selection. Invalid tokens get zero probability. This eliminates syntax errors and schema violations at the cost of latency (schema compilation on first request) and flexibility (no additionalProperties: true, no recursive schemas, limited anyOf/oneOf depth).

{
  "name": "extract_invoice",
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": {
      "vendor": { "type": "string" },
      "total_cents": { "type": "integer", "minimum": 0 },
      "line_items": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "description": { "type": "string" },
            "quantity": { "type": "integer" },
            "unit_price_cents": { "type": "integer" }
          },
          "required": ["description", "quantity", "unit_price_cents"],
          "additionalProperties": false
        }
      }
    },
    "required": ["vendor", "total_cents", "line_items"],
    "additionalProperties": false
  }
}

Anthropic’s tool-use schema validation happens post-generation. The model sees the schema in the system prompt and tries to comply, but nothing prevents it from emitting {"vendor": "Acme", "total_cents": "five thousand"} — a string where an integer belongs. You validate after the fact and retry on failure.

Gemini’s responseSchema with responseMimeType: "application/json" constrains the decoder similarly to OpenAI strict mode. The schema subset supports type, properties, required, items, enum, format (date-time, email, etc.), and nullable. It rejects additionalProperties, patternProperties, and complex composition keywords. First-request latency includes schema compilation.

import google.generativeai as genai

model = genai.GenerativeModel(
    "gemini-1.5-flash",
    generation_config=genai.GenerationConfig(
        response_mime_type="application/json",
        response_schema={
            "type": "object",
            "properties": {
                "vendor": {"type": "string"},
                "total_cents": {"type": "integer"},
                "line_items": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "description": {"type": "string"},
                            "quantity": {"type": "integer"},
                            "unit_price_cents": {"type": "integer"}
                        },
                        "required": ["description", "quantity", "unit_price_cents"]
                    }
                }
            },
            "required": ["vendor", "total_cents", "line_items"]
        }
    )
)
response = model.generate_content("Extract invoice from: ...")
print(response.text)  # guaranteed valid JSON matching schema

Error modes and retry strategies

OpenAI basic JSON mode (type: "json_object") fails silently on schema violations. You parse, validate, and retry with a corrected prompt. With strict: true, the API returns a 400 if the schema is unsupported; runtime violations are nearly impossible but you still handle finish_reason: "length" truncation.

Anthropic fails in three ways: (1) the model emits text instead of a tool call — check stop_reason == "tool_use"; (2) the tool call arguments fail JSON parse — rare but happens on long outputs; (3) arguments parse but violate schema — validate and retry. Each retry burns the full context window again.

Gemini returns finish_reason: "MALFORMED_FUNCTION_CALL" if the decoder cannot satisfy the schema (usually context overflow or unsupported schema feature). You also get SAFETY blocks that truncate output mid-structure. Both require retry with shorter context or relaxed schema.

Latency and throughput characteristics

OpenAI strict mode adds 200–800 ms on first request per unique schema (compilation cache is per-model, per-organization). Subsequent requests are comparable to non-structured generation. Basic JSON mode has no measurable overhead.

Anthropic tool use adds no infrastructure latency — the model just emits a tool call block. But you often need 2–3 retries for schema compliance on complex schemas, multiplying effective latency.

Gemini schema compilation happens on the first request per schema per session. Cold start adds 300–600 ms. Warm requests match unstructured latency. Long contexts (>100k tokens) increase compilation time proportionally.

Token accounting and cost

OpenAI counts schema tokens in the prompt (function definition) and output tokens in the generated JSON. Strict mode outputs tend to be more verbose (no optional field omission), slightly increasing output token count.

Anthropic counts the tool definition in the system prompt and the tool call block in output. The tool call wrapper ({"name": "...", "input": {...}}) adds ~30 tokens overhead per call.

Gemini counts the schema in the generation config (not in prompt tokens) and the JSON output normally. No wrapper overhead.

Streaming behavior

OpenAI streams JSON tokens incrementally with stream: true. You get partial JSON that parses only at completion. Use a streaming JSON parser (e.g., json-stream or orjson with iterative parsing) if you need progressive UI updates.

Anthropic streams tool call blocks as delta events. The input field arrives in chunks; you accumulate and parse at the end. No native progressive parsing support.

Gemini does not stream structured output as of the 1.5 releases. stream: true with responseMimeType: "application/json" returns the full JSON at once. This is a known limitation for high-latency UX.

Ecosystem and tooling

OpenAI has first-party SDK support for strict schemas in Python, Node, and Go. The pydantic integration (openai.beta.chat.completions.parse) auto-generates schemas from models and validates responses — the best developer experience of the three.

from openai import OpenAI
from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total_cents: int
    line_items: list[LineItem]

client = OpenAI()
completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[{"role": "user", "content": "Extract invoice..."}],
    response_format=Invoice,
)
invoice = completion.choices[0].message.parsed  # Invoice instance

Anthropic’s SDKs expose tool use but no schema-to-model mapping. Community libraries like instructor or pydantic-ai fill the gap with retry loops and validation.

Gemini’s Python SDK accepts dict schemas directly. No first-party Pydantic integration exists yet; you serialize models to dict manually.

Limits and constraints summary

Which to choose by use case

High-throughput extraction with strict contracts — OpenAI strict mode. The decoder guarantee eliminates validation retries, Pydantic integration cuts boilerplate, and streaming works for progressive UIs. Pay the cold-start compilation once per schema.

Prototyping and flexible schemas — Anthropic tool use. No schema compilation step, full JSON Schema expressiveness (even if advisory), and Claude’s reasoning quality often produces cleaner JSON on first try for ambiguous tasks. Accept the retry loop.

Long-context document processing with schema — Gemini 1.5 Pro/Flash. 1M–2M token context window with decoder-enforced structure beats chunking strategies. No streaming is the trade-off; batch your extractions.

Multi-provider gateway routing — If you abstract behind a single OpenAI-compatible endpoint (like n4n.ai), map each provider’s structured output to a common internal representation: normalize tool calls to JSON, strip wrapper objects, unify error codes. The gateway can also enforce fallback ordering — try strict mode first, degrade to best-effort with validation on failure.

Cost-sensitive high-volume — Compare per-1k-output-token prices for your model tier. OpenAI gpt-4o-mini strict mode is often cheapest for simple schemas. Gemini Flash wins on long-context unit economics. Anthropic Haiku competes on latency-sensitive workloads where retries are rare.

Pick one primary provider for your structured-output pipeline, optimize prompts and schemas for its enforcement model, and treat cross-provider portability as a migration target — not a design constraint.