JSON mode explained: how LLMs return structured data

JSON mode is a model-level constraint that forces an LLM to emit only valid JSON by restricting its token vocabulary during generation. Instead of hoping the model follows formatting instructions in the system prompt, the inference engine masks out every token that would produce syntactically invalid JSON at each decoding step. The result is guaranteed parseable output without post-processing or retry loops.

How JSON mode works under the hood

Most providers implement JSON mode through constrained decoding — a technique that intersects the model’s probability distribution with a formal grammar. At each generation step, the sampler computes the next-token probabilities as usual, then applies a mask derived from a JSON grammar (typically a context-free grammar or regular expression). Tokens that would violate JSON syntax — unmatched braces, trailing commas, unquoted keys, invalid escape sequences — receive probability zero and are renormalized out.

This happens inside the inference engine, not in the model weights. The model still “thinks” in natural language, but the decoder only allows tokens that keep the output on a valid JSON parse path. Some implementations go further: they accept a JSON Schema and compile it into a pushdown automaton that enforces not just syntax but structural validity — required fields, enum values, array length bounds, nested object shapes.

# Conceptual constrained decoding loop (simplified)
def constrained_decode(logits, grammar_state, schema_automaton):
    # Mask invalid next tokens per JSON grammar + schema
    valid_token_mask = schema_automaton.allowed_tokens(grammar_state)
    masked_logits = logits.masked_fill(~valid_token_mask, -float('inf'))
    next_token = sample(masked_logits)
    grammar_state = schema_automaton.transition(grammar_state, next_token)
    return next_token, grammar_state

OpenAI’s response_format: { "type": "json_object" } uses a grammar-only approach — it guarantees syntactic validity but not schema conformance. Anthropic’s “tool use” and OpenAI’s “structured outputs” (the strict: true variant) compile JSON Schema into a deterministic finite automaton for full validation. The distinction matters: grammar-only mode can still emit {"user_id": "not-an-integer"} while schema-constrained mode rejects it at decode time.

Why engineers reach for JSON mode

Unstructured text is a liability in production pipelines. Before JSON mode, teams relied on prompt engineering (“return only valid JSON, no markdown, no commentary”) and retry logic with exponential backoff. Failure rates of 5–15% were common even with careful prompting, especially on smaller models or complex schemas. Each failure meant a full regeneration — doubling latency and token cost.

JSON mode eliminates the syntax failure class entirely. The model cannot emit a trailing comma, a stray newline, or a Python-style True instead of true. This shifts the failure mode from “invalid JSON” to “valid JSON that doesn’t match your schema” — a narrower, more debuggable problem.

The trade-off is latency. Constrained decoding adds per-token overhead: the automaton must be consulted at every step. On typical hardware this adds 5–15% wall-clock time versus unconstrained generation. For high-throughput workloads, that compounds. Some teams mitigate this by using JSON mode only for the final extraction step in a chain — letting a cheaper, faster model draft the response in natural language, then a second pass with JSON mode structures it.

Concrete example: extracting structured data from invoices

Consider a document-processing pipeline that ingests PDF invoices and emits normalized records for an ERP system. Without JSON mode, the prompt might include:

Extract the following fields as JSON: vendor_name, invoice_number, date, line_items (array of {description, quantity, unit_price, total}), subtotal, tax, total.
Return ONLY valid JSON. No markdown. No explanation.

Even with that instruction, a model might emit:

{
  "vendor_name": "Acme Corp",
  "invoice_number": "INV-2024-001",
  "date": "2024-01-15",
  "line_items": [
    {"description": "Widget A", "quantity": 10, "unit_price": 25.00, "total": 250.00},
    {"description": "Widget B", "quantity": 5, "unit_price": 50.00, "total": 250.00}
  ],
  "subtotal": 500.00,
  "tax": 40.00,
  "total": 540.00
}

But it might also emit:

{
  "vendor_name": "Acme Corp",
  "invoice_number": "INV-2024-001",
  "date": "January 15, 2024",  // wrong format
  "line_items": [
    {"description": "Widget A", "quantity": 10, "unit_price": 25.00, "total": 250.00},
    {"description": "Widget B", "quantity": 5, "unit_price": 50.00, "total": 250.00},  // trailing comma
  ],
  "subtotal": 500.00,
  "tax": 40.00,
  "total": 540.00
}  // trailing comma

Or wrap it in markdown fences. Or preface with “Here is the JSON:”. JSON mode eliminates the syntax errors. Schema-constrained mode (strict structured outputs) also rejects the date format mismatch if your schema specifies format: "date".

{
  "type": "object",
  "properties": {
    "vendor_name": { "type": "string" },
    "invoice_number": { "type": "string", "pattern": "^INV-\\d{4}-\\d{3}$" },
    "date": { "type": "string", "format": "date" },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "description": { "type": "string" },
          "quantity": { "type": "integer", "minimum": 1 },
          "unit_price": { "type": "number", "minimum": 0 },
          "total": { "type": "number", "minimum": 0 }
        },
        "required": ["description", "quantity", "unit_price", "total"],
        "additionalProperties": false
      }
    },
    "subtotal": { "type": "number", "minimum": 0 },
    "tax": { "type": "number", "minimum": 0 },
    "total": { "type": "number", "minimum": 0 }
  },
  "required": ["vendor_name", "invoice_number", "date", "line_items", "subtotal", "tax", "total"],
  "additionalProperties": false
}

With strict mode enabled, the decoder will not emit "date": "January 15, 2024" — the automaton has no transition for that token sequence given the format: "date" constraint. The model is forced to produce 2024-01-15 or fail (which in practice means the generation halts or the provider returns an error).

JSON mode versus function calling versus structured outputs

These three terms get conflated. They are distinct mechanisms:

Function calling adds a routing layer: the model selects a function name and fills its arguments. The arguments are JSON-mode-constrained to the function’s parameter schema. Structured outputs generalize this — you pass a JSON Schema directly without wrapping it in a function definition. Under the hood, both use the same constrained-decoding machinery.

If you only need “give me this shape,” structured outputs (strict JSON Schema) is the cleanest abstraction. Function calling belongs when the model must decide among multiple operations.

Common misconceptions

“JSON mode makes the model smarter”

It doesn’t. The model’s reasoning capacity is unchanged. Constrained decoding only prunes the output space. A model that hallucinates a vendor name in unconstrained mode will still hallucinate it in JSON mode — the output will just be valid JSON containing the hallucination. Schema constraints catch type/format errors, not factual ones.

“I don’t need validation if I use JSON mode”

You still need application-level validation. JSON mode guarantees syntax; strict structured outputs guarantee schema conformance. Neither guarantees business logic — that total == subtotal + tax, that invoice_number exists in your database, that line_items aren’t duplicates. Parse, then validate.

“JSON mode works the same across all providers”

It doesn’t. OpenAI’s json_object is grammar-only. Anthropic’s tool use constrains to the function schema. Google’s response_mime_type: "application/json" is grammar-only. OpenAI’s strict: true structured outputs compile JSON Schema to an automaton. Mistral’s format: "json" is grammar-only. The behavior when the model wants to emit invalid JSON differs: some providers return an error, some truncate, some fall back to best-effort. Test your specific provider.

“Schema-constrained mode supports all JSON Schema features”

Most implementations support a subset. additionalProperties: false, required, enum, const, type, items, properties, minimum/maximum/minLength/maxLength, pattern, format (date, email, uuid) are widely supported. oneOf, anyOf, allOf, not, if/then/else, recursive references ($ref cycles), and dependentSchemas often are not. Check the provider’s documentation before designing complex schemas.

“Streaming works the same with JSON mode”

Streaming + constrained decoding is tricky. The automaton state must be maintained across chunks. Some providers buffer until a complete valid object is ready (defeating streaming’s latency benefit). Others stream token-by-token but may emit partial JSON that isn’t parseable until complete. If you need streaming, verify the provider’s behavior — and consider whether you actually need streaming for structured extraction (usually you don’t; the whole object arrives in one shot anyway).

When to skip JSON mode

JSON mode adds latency and restricts the model’s expressiveness. Avoid it when:

• The output is genuinely free-form (summaries, creative writing, chat)
• You’re prototyping and the schema changes daily — prompt engineering is faster to iterate
• The model is too small to reliably follow the schema even with constraints (sub-7B models often degrade noticeably under strict decoding)
• You need the model to “think out loud” before producing JSON — use a two-step chain instead: reasoning in step 1, JSON mode in step 2

Practical tips

Put the schema in the system prompt even with strict mode. The model still needs to know the schema to generate semantically correct values. The automaton only blocks invalid tokens; it doesn’t guide the model toward good ones. A system message like “You are an invoice extractor. Output JSON matching this schema: {…}” improves field-level accuracy significantly.

Use additionalProperties: false everywhere. Without it, the model can emit extra fields that your downstream code ignores — until one of them collides with a future field name. Strict mode enforces this at decode time.

Prefer enum over free strings for categorical fields. {"type": "string", "enum": ["pending", "paid", "overdue"]} is both more constrained and more efficient for the automaton than a pattern or description.

Test with temperature > 0. Constrained decoding interacts with sampling. At temperature 0 the model is deterministic; at 0.7 it may hit the automaton’s constraints more often, causing renormalization that distorts the intended distribution. Verify your actual production temperature setting.

Log the raw completion tokens on failure. When structured output errors occur (rare with strict mode, but possible with provider bugs or schema limits), the raw token stream is the only way to debug whether the model or the automaton misbehaved.

Summary

JSON mode moves JSON validity from a prompt-engineering hope to a decode-time guarantee. Grammar-only mode (OpenAI json_object, Mistral format: "json", Google response_mime_type) catches syntax errors. Schema-constrained mode (OpenAI strict: true, Anthropic tool use) catches structural errors too. Neither catches semantic errors — you still validate.

For production extraction pipelines, schema-constrained structured outputs are the right default. For ad-hoc tasks or models without strict support, grammar-only JSON mode is a meaningful improvement over prompt-only approaches. Function calling wraps the same machinery for agent workflows where the model chooses the operation.

The constraint is in the decoder, not the model. Treat it as a runtime guarantee, not a capability upgrade.