6 common JSON mode failures and how to fix them

JSON mode failures fix is a recurring task for engineers shipping LLM agents, because the distance between a declared schema and the bytes a model actually emits is larger than vendor docs imply. This article breaks down six concrete failure modes I have hit in production and the minimal changes that made them disappear.

1. Strict mode left off, schema treated as a suggestion

Most OpenAI-compatible endpoints accept response_format with a JSON schema, but many default to non-strict adherence. The model will happily add fields you never declared or silently drop a required key when it thinks the text flows better without it.

The fix is to pass strict: true and set additionalProperties: false at every object level. Strict mode forces the model to constrain output to the schema; if it cannot, it errors instead of emitting garbage.

from openai import OpenAI

client = OpenAI()
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Return a user object"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "strict": True,
            "name": "user",
            "schema": {
                "type": "object",
                "additionalProperties": False,
                "properties": {
                    "id": {"type": "integer"},
                    "name": {"type": "string"}
                },
                "required": ["id", "name"]
            }
        }
    }
)
print(resp.choices[0].message.content)

If you skip strict, you are relying on the model’s manners, not its parser. Treat strict mode as mandatory for any agent that feeds the output into code.

2. Truncated output from a too-small max_tokens

JSON mode does not exempt you from token limits. A complex nested object can blow past a conservative max_tokens setting, leaving you with { "id": 12, "name": "Ja and a JSONDecodeError.

Calculate a rough upper bound from your schema before the call. For a response with arrays, multiply expected item count by per-item token estimate. Set max_tokens with headroom, then handle truncation explicitly on the client.

import json

try:
    obj = json.loads(completion)
except json.JSONDecodeError:
    # retry with higher limit or request a smaller slice
    retry_with_max_tokens(current_limit * 2)

Streaming helps: accumulate chunks and if the stream ends without a closing brace, you know you truncated before parsing. Never assume a 200 response contains valid JSON just because the API accepted your schema.

3. Markdown fences leaking around the payload

Several open-weight models and some proxies ignore response_format and wrap the JSON in json … . Your json.loads then fails on the first backtick. This is especially common when the same endpoint serves both chat and structured requests.

Strip fences defensively. A small normalization step costs nothing and saves a retry round-trip.

import re

def extract_json(text: str) -> str:
    match = re.search(r"```(?:json)?\s*([\s\S]*?)```", text)
    if match:
        return match.group(1).strip()
    return text.strip()

obj = json.loads(extract_json(raw_response))

If you route through a gateway that honors client routing directives, you can pin a model known to respect JSON mode rather than hoping the fallback behaves. Either way, parse the wrapped text, don’t trust the wrapper.

4. Schema uses keywords the model backend ignores

JSON Schema is a large spec; LLM providers implement a subset. I have seen $ref, pattern, format, and minimum silently dropped. The model then emits "email": "not-an-email" because the constraint was never enforced.

Flatten the schema. Inline definitions instead of $ref. Drop pattern and validate post-hoc in code. Keep only type, enum, properties, required, additionalProperties, and items for arrays.

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "status": {"type": "string", "enum": ["open", "closed"]}
  },
  "required": ["status"]
}

After parsing, run a real validator (e.g., jsonschema in Python) if you need those extra constraints. The model’s “JSON mode” is a shape enforcer, not a contract tester.

5. Type coercion masking malformed output

Python’s json.loads will decode "id": "42" as a string, and your downstream code may blindly cast it. That hides the fact the model violated the integer declaration. Over time, a loosely typed pipeline accumulates silent corruptions.

Use a validation layer that rejects wrong types. Pydantic is the pragmatic choice.

from pydantic import BaseModel, ValidationError

class User(BaseModel):
    id: int
    name: str

try:
    user = User(**json.loads(raw))
except ValidationError as e:
    # route to fix or reject
    log_and_retry(e)

This turns a latent JSON mode failure into a visible exception. The fix is not smarter prompts; it is refusing to accept non-conforming bytes.

6. Provider fallback swaps the model mid-request

When a primary provider is rate-limited, some gateways fail over to a secondary model. A gateway such as n4n.ai forwards your schema and cache-control hints, but the backup model may not support strict mode or may interpret enum loosely. Your parser suddenly sees new keys or missing fields.

Pin the model explicitly if consistency matters, or add a post-parse assertion that runs regardless of which model answered.

assert set(obj.keys()) == {"id", "name"}, "schema drift after fallback"

If you rely on automatic fallback for uptime, treat the response as untrusted input from a different vendor. Validate against the schema you sent, not the one you assume came back.

Summary

JSON mode failures fix is mostly about refusing to trust the model’s output until code has verified it. The schema is a hint to the sampler; your parser is the contract.