JSON mode promises structured output but frequently delivers syntax errors, schema violations, and silent truncation. When you need to debug JSON mode errors LLM pipelines produce, the failure modes are predictable once you know where to look. This guide walks through the most common failure patterns, shows how to isolate each one, and gives you verification steps you can run in CI.
Step 1: Capture the raw response before any parsing
The first mistake engineers make is wrapping the API call in a try/except that swallows the raw text. You cannot debug what you cannot see. Always log or return the complete response object — including headers, finish reason, and the unparsed text — before any JSON parser touches it.
import json
import os
from openai import OpenAI
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
def call_json_mode(prompt: str, model: str = "gpt-4o-mini") -> dict:
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0,
max_tokens=2000,
)
# Return everything you need for debugging
return {
"raw_text": resp.choices[0].message.content,
"finish_reason": resp.choices[0].finish_reason,
"usage": resp.usage.model_dump() if resp.usage else None,
"model": resp.model,
"id": resp.id,
}
# Example usage
result = call_json_mode("Return a JSON object with keys: name, age, is_student")
print(json.dumps(result, indent=2))Verify success: The raw_text field contains valid JSON that json.loads() parses without exception. The finish_reason is "stop", not "length" or "content_filter".
Step 2: Check for truncation caused by token limits
A finish_reason of "length" means the model hit max_tokens mid-generation. The output will be syntactically incomplete — missing closing braces, trailing commas, or cut-off strings. This is the single most common cause of JSON parse failures in production.
def diagnose_truncation(result: dict) -> dict:
raw = result["raw_text"]
finish = result["finish_reason"]
issues = []
if finish == "length":
issues.append("TRUNCATED: finish_reason is 'length'")
# Heuristic: count unmatched braces/brackets
open_braces = raw.count("{")
close_braces = raw.count("}")
open_brackets = raw.count("[")
close_brackets = raw.count("]")
if open_braces != close_braces:
issues.append(f"UNBALANCED BRACES: {open_braces} open vs {close_braces} close")
if open_brackets != close_brackets:
issues.append(f"UNBALANCED BRACKETS: {open_brackets} open vs {close_brackets} close")
# Check for incomplete string literals (odd number of unescaped quotes)
# Simple heuristic: count quotes not preceded by backslash
import re
unescaped_quotes = len(re.findall(r'(?<!\\)"', raw))
if unescaped_quotes % 2 != 0:
issues.append("ODD QUOTE COUNT: likely unterminated string")
return {"issues": issues, "raw_preview": raw[:500]}
# Test with a deliberately low token limit
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Generate a JSON array of 1000 objects with id and name"}],
response_format={"type": "json_object"},
max_tokens=100, # Too low
)
result = {"raw_text": resp.choices[0].message.content, "finish_reason": resp.choices[0].finish_reason}
print(json.dumps(diagnose_truncation(result), indent=2))Fix: Increase max_tokens to at least 1.5× your expected output size. For variable-length outputs, set a generous ceiling (4000–8000) and monitor actual usage. Verify success: finish_reason returns "stop" and brace/bracket counts balance.
Step 3: Validate against a JSON Schema, not just syntax
Syntactically valid JSON can still violate your contract — missing required fields, wrong types, enum violations. Use a schema validator (Pydantic, jsonschema) to catch these before they propagate.
from pydantic import BaseModel, ValidationError, Field
from typing import Literal
import json
class UserProfile(BaseModel):
name: str = Field(min_length=1, max_length=100)
age: int = Field(ge=0, le=150)
is_student: bool
tier: Literal["free", "pro", "enterprise"]
tags: list[str] = Field(default_factory=list, max_length=20)
def validate_against_schema(raw_text: str, schema_class: type[BaseModel]) -> dict:
try:
data = json.loads(raw_text)
instance = schema_class(**data)
return {"valid": True, "data": instance.model_dump()}
except json.JSONDecodeError as e:
return {"valid": False, "error_type": "syntax", "message": str(e), "pos": e.pos}
except ValidationError as e:
errors = []
for err in e.errors():
loc = " -> ".join(str(x) for x in err["loc"])
errors.append(f"{loc}: {err['msg']} (type={err['type']})")
return {"valid": False, "error_type": "schema", "errors": errors}
# Test with valid and invalid outputs
valid_json = '{"name": "Ada", "age": 36, "is_student": false, "tier": "pro", "tags": ["python", "compilers"]}'
invalid_json = '{"name": "", "age": -5, "is_student": "yes", "tier": "premium", "tags": "not-a-list"}'
print("Valid:", json.dumps(validate_against_schema(valid_json, UserProfile), indent=2))
print("Invalid:", json.dumps(validate_against_schema(invalid_json, UserProfile), indent=2))Verify success: validate_against_schema returns {"valid": true, "data": {...}} for at least 10 consecutive requests with your production prompts.
Step 4: Handle the “markdown code fence” leak
Even with response_format: {type: "json_object"}, some models (especially older ones or non-OpenAI providers) wrap output in ```json fences. This breaks strict parsers. Strip fences defensively.
import re
def strip_markdown_fences(text: str) -> str:
"""Remove ```json ... ``` or ``` ... ``` wrappers if present."""
# Match ```json\n{...}\n``` or ```\n{...}\n```
pattern = r'^```(?:json)?\s*\n?(.*?)\n?```$'
match = re.match(pattern, text.strip(), re.DOTALL)
if match:
return match.group(1).strip()
return text.strip()
# Test cases
test_cases = [
'{"valid": true}',
'```json\n{"wrapped": true}\n```',
'```\n{"also_wrapped": true}\n```',
' ```json\n{"spaced": true}\n``` ',
]
for tc in test_cases:
print(f"Input: {tc[:50]}... -> Output: {strip_markdown_fences(tc)}")Verify success: strip_markdown_fences returns parseable JSON for all four test cases above.
Step 5: Detect and repair common syntax errors automatically
When you control the prompt but not the model (e.g., routing across providers), you will see recurring syntax errors: trailing commas, single quotes, unquoted keys, Python True/False/None instead of true/false/null. A lightweight repair step can salvage many responses.
def repair_common_json_errors(text: str) -> str:
"""Best-effort repair for frequent LLM JSON mistakes."""
# 1. Strip markdown fences first
text = strip_markdown_fences(text)
# 2. Replace Python literals with JSON literals
text = re.sub(r'\bTrue\b', 'true', text)
text = re.sub(r'\bFalse\b', 'false', text)
text = re.sub(r'\bNone\b', 'null', text)
# 3. Fix single-quoted strings (naive but catches common cases)
# Only replace single quotes that appear to delimit strings, not apostrophes
# This regex finds '...' not preceded/followed by word chars
text = re.sub(r"(?<!\w)'(.*?)'(?!\w)", r'"\1"', text)
# 4. Remove trailing commas before } or ]
text = re.sub(r',\s*([}\]])', r'\1', text)
# 5. Fix unquoted keys (naive: word followed by colon at start of line or after {,)
text = re.sub(r'([{,])\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*:', r'\1"\2":', text)
return text
def parse_with_repair(raw_text: str) -> dict:
"""Try parse, then repair, then parse again."""
for attempt, txt in enumerate([raw_text, repair_common_json_errors(raw_text)]):
try:
return {"success": True, "data": json.loads(txt), "repaired": attempt == 1}
except json.JSONDecodeError as e:
if attempt == 1:
return {"success": False, "error": str(e), "repaired_text": txt[:200]}
return {"success": False, "error": "unreachable"}
# Test repairs
broken_cases = [
"{'name': 'test', 'active': True, 'count': None,}", # single quotes, Python literals, trailing comma
'{name: "unquoted key", value: 42}', # unquoted keys
'```json\n{"wrapped": True,}\n```', # fence + Python literal + trailing comma
]
for bc in broken_cases:
print(json.dumps(parse_with_repair(bc), indent=2))Verify success: Your repair function handles the top 5 error patterns you observe in logs. Track repaired: true rate — if it exceeds 5%, fix the prompt or switch models instead of relying on repair.
Step 6: Enforce schema adherence through prompt engineering
The most reliable fix is preventing errors upstream. Use few-shot examples that demonstrate exact formatting, and include the schema in the system prompt.
SYSTEM_PROMPT = """You are a JSON API. Output ONLY valid JSON matching this schema:
{
"type": "object",
"properties": {
"name": {"type": "string", "minLength": 1, "maxLength": 100},
"age": {"type": "integer", "minimum": 0, "maximum": 150},
"is_student": {"type": "boolean"},
"tier": {"type": "string", "enum": ["free", "pro", "enterprise"]},
"tags": {"type": "array", "items": {"type": "string"}, "maxItems": 20}
},
"required": ["name", "age", "is_student", "tier"],
"additionalProperties": false
}
Examples:
User: "Ada, 36, not a student, pro tier, likes python and compilers"
Assistant: {"name": "Ada", "age": 36, "is_student": false, "tier": "pro", "tags": ["python", "compilers"]}
User: "Bob, 22, student, free tier, no tags"
Assistant: {"name": "Bob", "age": 22, "is_student": true, "tier": "free", "tags": []}
"""
def call_with_schema_prompt(user_input: str) -> dict:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_input},
],
response_format={"type": "json_object"},
temperature=0,
max_tokens=1000,
)
return {
"raw_text": resp.choices[0].message.content,
"finish_reason": resp.choices[0].finish_reason,
}
# Test
result = call_with_schema_prompt("Carol, 28, student, enterprise tier, tags: rust, wasm, systems")
print(json.dumps(validate_against_schema(result["raw_text"], UserProfile), indent=2))Verify success: Run 50 representative inputs through this prompt. Schema validation passes on all 50 with zero repairs needed.
Step 7: Build a debugging harness for CI/CD
Automate the above checks so regressions fail fast. This harness runs a test suite of prompts and asserts validity, latency, and token usage.
import time
from dataclasses import dataclass
from typing import Callable
@dataclass
class TestCase:
name: str
prompt: str
validator: Callable[[str], dict] # returns {"valid": bool, ...}
max_latency_ms: int = 5000
max_tokens: int = 2000
TEST_CASES = [
TestCase(
name="basic_profile",
prompt="Generate a user profile for a 25-year-old developer named Alex, pro tier",
validator=lambda t: validate_against_schema(t, UserProfile),
),
TestCase(
name="student_free_tier",
prompt="Student user, age 19, free tier, tags: learning, python",
validator=lambda t: validate_against_schema(t, UserProfile),
),
TestCase(
name="edge_case_max_tags",
prompt="Enterprise user with 20 tags: tag1 through tag20",
validator=lambda t: validate_against_schema(t, UserProfile),
),
]
def run_harness(model: str = "gpt-4o-mini") -> dict:
results = []
for tc in TEST_CASES:
start = time.perf_counter()
resp = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": tc.prompt},
],
response_format={"type": "json_object"},
temperature=0,
max_tokens=tc.max_tokens,
)
latency_ms = (time.perf_counter() - start) * 1000
raw = resp.choices[0].message.content
validation = tc.validator(raw)
passed = (
validation.get("valid", False)
and latency_ms <= tc.max_latency_ms
and resp.choices[0].finish_reason == "stop"
)
results.append({
"test": tc.name,
"passed": passed,
"latency_ms": round(latency_ms, 1),
"finish_reason": resp.choices[0].finish_reason,
"tokens": resp.usage.total_tokens if resp.usage else None,
"validation": validation,
})
all_passed = all(r["passed"] for r in results)
return {"all_passed": all_passed, "results": results}
# Run it
harness_result = run_harness()
print(json.dumps(harness_result, indent=2))
# In CI, exit non-zero on failure
if not harness_result["all_passed"]:
exit(1)Verify success: The harness passes in CI on every merge. Add new test cases whenever you discover a new failure pattern in production logs.
Step 8: Monitor provider-specific quirks in production
If you route across multiple providers (OpenAI, Anthropic, open-source models via inference endpoints), each has distinct JSON mode behaviors. Log the model identifier alongside every failure to spot provider-specific patterns.
# Example: structured logging for production observability
import logging
import uuid
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("json_mode")
def log_json_attempt(prompt: str, model: str, result: dict, validation: dict, request_id: str = None):
request_id = request_id or str(uuid.uuid4())[:8]
logger.info(
"json_mode_attempt",
extra={
"request_id": request_id,
"model": model,
"prompt_hash": hash(prompt) % 1000000,
"finish_reason": result.get("finish_reason"),
"latency_ms": result.get("latency_ms"),
"tokens_in": result.get("usage", {}).get("prompt_tokens"),
"tokens_out": result.get("usage", {}).get("completion_tokens"),
"valid": validation.get("valid"),
"error_type": validation.get("error_type"),
"repaired": validation.get("repaired", False),
}
)
# Simulated production call with routing
def production_call(prompt: str, model: str) -> dict:
start = time.perf_counter()
resp = client.chat.completions.create(
model=model,
messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0,
max_tokens=2000,
)
latency_ms = (time.perf_counter() - start) * 1000
raw = resp.choices[0].message.content
# Parse with repair
parse_result = parse_with_repair(raw)
if parse_result["success"]:
validation = validate_against_schema(parse_result["data"], UserProfile)
validation["repaired"] = parse_result.get("repaired", False)
else:
validation = {"valid": False, "error_type": "syntax", "message": parse_result["error"]}
result = {
"raw_text": raw,
"finish_reason": resp.choices[0].finish_reason,
"latency_ms": latency_ms,
"usage": resp.usage.model_dump() if resp.usage else None,
}
log_json_attempt(prompt, model, result, validation)
return {"result": result, "validation": validation, "parsed": parse_result.get("data")}
# Test with two different models (if available)
for m in ["gpt-4o-mini", "gpt-3.5-turbo"]:
try:
out = production_call("User: Dana, 31, not student, enterprise, tags: kubernetes, go", m)
print(f"{m}: valid={out['validation'].get('valid')}, repaired={out['validation'].get('repaired')}")
except Exception as e:
print(f"{m}: ERROR - {e}")Verify success: Your logs show <1% valid: false rate per model. Any model exceeding 2% gets a dedicated test case in the harness (Step 7) and a prompt fix.
Step 9: Use structured outputs (function calling) when JSON mode isn’t enough
OpenAI’s response_format: {type: "json_schema"} (structured outputs) enforces schema at the tokenizer level, eliminating syntax errors entirely. Migrate high-stakes paths to this mode.
# Requires openai>=1.10.0 and a model that supports structured outputs (gpt-4o-2024-08-06+)
from pydantic import BaseModel
class StrictUserProfile(BaseModel):
name: str
age: int
is_student: bool
tier: Literal["free", "pro", "enterprise"]
tags: list[str]
def call_structured_outputs(prompt: str, model: str = "gpt-4o-2024-08-06") -> dict:
resp = client.beta.chat.completions.parse(
model=model,
messages=[
{"role": "system", "content": "Extract user profile."},
{"role": "user", "content": prompt},
],
response_format=StrictUserProfile,
temperature=0,
)
# resp.choices[0].message.parsed is already a Pydantic model
parsed = resp.choices[0].message.parsed
return {
"valid": True,
"data": parsed.model_dump() if parsed else None,
"finish_reason": resp.choices[0].finish_reason,
"refusal": resp.choices[0].message.refusal,
}
# Test
try:
result = call_structured_outputs("Eve, 42, enterprise, tags: leadership, strategy")
print(json.dumps(result, indent=2, default=str))
except Exception as e:
print(f"Structured outputs not available: {e}")Verify success: Zero syntax errors in 1,000 requests. refusal field captures the rare cases where the model cannot comply.
Summary checklist
When you debug JSON mode errors LLM pipelines surface, follow this order:
- Log raw response — never parse before capturing
finish_reasonand full text - Check truncation —
finish_reason: "length"means raisemax_tokens - Validate schema — syntax ≠ correctness; use Pydantic or jsonschema
- Strip fences — defensive cleanup for provider inconsistencies
- Repair common errors — trailing commas, Python literals, single quotes, unquoted keys
- Fix the prompt — few-shot examples + schema in system prompt prevents 90% of errors
- Automate in CI — harness with latency, token, and validity assertions
- Monitor by model — route-around or prompt-tune problematic providers
- Upgrade to structured outputs — tokenizer-enforced schemas for critical paths
Each step has a verification criterion. Implement them incrementally; the harness (Step 7) is the forcing function that keeps regressions out of production.