Getting a language model to return clean, parseable data is half the battle. To enforce json schema llm response in production, you need more than a hopeful prompt—you need the API’s structured output support, a strict schema, and a validation layer that rejects nonconforming payloads. This guide walks through the exact steps to lock down model outputs to a contract your code can trust.
Step 1: Define a strict JSON Schema
Start by writing a JSON Schema that describes the exact shape you expect. For OpenAI-style structured outputs, the schema must have "additionalProperties": false at every object level and list all fields as required. Optional fields are not allowed in strict mode; if you need optionality, model it with nullable types and explicit null acceptance.
Below is a schema for extracting line items from an invoice:
{
"type": "object",
"additionalProperties": false,
"required": ["invoice_id", "date", "total", "line_items"],
"properties": {
"invoice_id": { "type": "string" },
"date": { "type": "string", "format": "date" },
"total": { "type": "number" },
"line_items": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["sku", "quantity", "price"],
"properties": {
"sku": { "type": "string" },
"quantity": { "type": "integer" },
"price": { "type": "number" }
}
}
}
}
}Keep the schema shallow where possible. Deeply nested unions increase the chance the model emits something the parser rejects. If you must represent polymorphism, use a discriminator field and a set of flat sub-schemas validated in your own code after the model returns.
Step 2: Send the schema to the model
Pass the schema via response_format of type json_schema. The strict: true flag tells the provider to constrain decoding to the schema. The OpenAI Python SDK accepts this directly:
from openai import OpenAI
client = OpenAI() # or point base_url at a gateway
resp = client.chat.completions.create(
model="gpt-4o-2024-08-06",
messages=[
{"role": "system", "content": "Extract invoice data as JSON."},
{"role": "user", "content": open("invoice.txt").read()},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "invoice",
"strict": True,
"schema": {
"type": "object",
"additionalProperties": False,
"required": ["invoice_id", "date", "total", "line_items"],
"properties": {
"invoice_id": {"type": "string"},
"date": {"type": "string", "format": "date"},
"total": {"type": "number"},
"line_items": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": False,
"required": ["sku", "quantity", "price"],
"properties": {
"sku": {"type": "string"},
"quantity": {"type": "integer"},
"price": {"type": "number"},
},
},
},
},
},
},
},
)
raw = resp.choices[0].message.contentIf you route through n4n.ai, its OpenAI-compatible endpoint forwards the same response_format block to any of the 240+ models behind it and automatically falls back when a provider is rate-limited or degraded, so the schema enforcement logic stays identical across backends.
Not every model supports strict. For those that only offer response_format={"type": "json_object"}, the model returns valid JSON but not necessarily your schema. You must validate downstream.
Step 3: Validate the returned payload
Never trust the model completely. Even with strict mode, provider bugs or version drift can leak an extra key. Run the parsed object through jsonschema before it touches your business logic:
import json
from jsonschema import validate, ValidationError
schema = { # same dict as Step 1
"type": "object",
"additionalProperties": False,
"required": ["invoice_id", "date", "total", "line_items"],
"properties": {
"invoice_id": {"type": "string"},
"date": {"type": "string", "format": "date"},
"total": {"type": "number"},
"line_items": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": False,
"required": ["sku", "quantity", "price"],
"properties": {
"sku": {"type": "string"},
"quantity": {"type": "integer"},
"price": {"type": "number"},
},
},
},
},
}
def parse_invoice(raw: str) -> dict:
try:
data = json.loads(raw)
except json.JSONDecodeError as e:
raise ValueError("model did not return JSON") from e
try:
validate(instance=data, schema=schema)
except ValidationError as e:
raise ValueError(f"schema violation: {e.message}") from e
return dataThe format: date check is not enforced by default in jsonschema without a formatter. Add FormatChecker if you care about RFC 3339 compliance:
from jsonschema import Draft202012Validator, FormatChecker
Draft202012Validator(schema, format_checker=FormatChecker()).validate(data)Step 4: Wrap the call in a retry and fallback loop
A single call can fail three ways: network error, JSON decode error, or schema violation. Build a thin client that retries with the same schema and switches model only if the provider is consistently failing. Because you already enforce json schema llm response at the protocol level, the fallback model must also accept the schema.
import time
def extract_with_retry(text: str, attempts: int = 3) -> dict:
last_err = None
for i in range(attempts):
try:
raw = call_model(text) # your Step 2 function
return parse_invoice(raw)
except ValueError as e:
last_err = e
time.sleep(2 ** i)
raise RuntimeError(f"failed after {attempts} attempts: {last_err}")If you use a gateway that honors client routing directives, you can pin a primary model and let the gateway shift to a secondary when the primary is over quota. The schema passes through untouched, so your validation layer never changes.
Step 5: Enforce the contract at your API boundary
The final step is to convert the validated dict into a typed object and reject anything that does not conform before it enters your system. Pydantic gives you a second line of defense and clear error messages:
from pydantic import BaseModel
class LineItem(BaseModel):
sku: str
quantity: int
price: float
class Invoice(BaseModel):
invoice_id: str
date: str
total: float
line_items: list[LineItem]
def to_invoice(data: dict) -> Invoice:
return Invoice(**data) # raises pydantic.ValidationError on mismatchChain it: invoice = to_invoice(parse_invoice(raw)). Now the rest of your codebase receives an Invoice instance, not a loose dict.
How to verify success
Write a test that proves the pipeline enforces the schema. Use a fixed prompt and a mocked completion that returns a correct payload, then a second test where the mock returns a payload missing total:
def test_happy_path(monkeypatch):
monkeypatch.setattr("__main__.call_model", lambda t: '{"invoice_id":"INV-1","date":"2024-01-01","total":10.0,"line_items":[]}')
inv = to_invoice(parse_invoice(extract_with_retry("dummy")))
assert inv.invoice_id == "INV-1"
def test_schema_violation(monkeypatch):
monkeypatch.setattr("__main__.call_model", lambda t: '{"invoice_id":"INV-1"}')
try:
extract_with_retry("dummy")
assert False, "should have raised"
except RuntimeError:
passRun pytest. If both pass, you enforce json schema llm response end to end. In production, watch logs for ValueError spikes—those indicate a model or provider regression, not a bug in your code.
Keep the schema in version control next to the validation code. When the downstream consumer needs a new field, change the schema, the Pydantic model, and the tests together. That discipline is what makes structured LLM outputs safe to ship.