Getting reliable JSON from an LLM used to mean prompt engineering and hopeful parsing. A structured outputs unified gateway lets you declare a schema once and route to any compliant backend, but the contract still lives in your code, not the model. This guide walks a concrete path from schema to production call.
1. Choose the right constraint level
JSON mode and structured outputs are not the same feature. JSON mode only guarantees the response parses as JSON. Structured outputs enforce a supplied schema, usually via response_format with a JSON schema and strictness flags.
Use JSON mode when you trust the model to fill arbitrary shapes and just need syntactic validity. Use structured outputs when downstream code will break on missing fields or wrong types.
# JSON mode: valid JSON, no schema check
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Return a user object."}],
response_format={"type": "json_object"}
)# Structured outputs: schema enforced
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Return a user."}],
response_format={
"type": "json_schema",
"json_schema": {
"name": "user",
"strict": True,
"schema": {
"type": "object",
"properties": {
"id": {"type": "integer"},
"name": {"type": "string"}
},
"required": ["id", "name"],
"additionalProperties": False
}
}
}
)Tradeoff: strict structured outputs reduce model freedom and can increase latency. Some providers reject schemas with deep nesting or unsupported keywords.
2. Write a schema that survives provider differences
Provider implementations of structured outputs diverge. OpenAI supports strict: true with limitations: top-level must be an object, all properties required, additionalProperties: false, no anyOf/oneOf in some versions. Anthropic’s tool-use coercion differs. A structured outputs unified gateway forwards your schema but cannot magically harmonize incompatible backends.
Keep schemas flat. Avoid regex patterns, format: date-time, and recursive references unless you’ve verified the target model supports them.
{
"type": "object",
"properties": {
"order_id": {"type": "string"},
"total_cents": {"type": "integer"},
"status": {"type": "string", "enum": ["pending", "paid", "refunded"]}
},
"required": ["order_id", "total_cents", "status"],
"additionalProperties": false
}Pitfall: omitting additionalProperties: false will fail strict validation on most gateways. Another: using nullable instead of type: ["string", "null"] breaks older parsers.
3. Send the request through the gateway
Point your existing OpenAI-compatible client at the gateway endpoint. A structured outputs unified gateway like n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models, so the same response_format block works whether the backend is OpenAI, Anthropic, or a local model.
from openai import OpenAI
client = OpenAI(
base_url="https://api.n4n.ai/v1",
api_key="YOUR_KEY"
)
resp = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "system", "content": "Output JSON only."},
{"role": "user", "content": "Describe order #42 for $19.99 paid."}],
response_format={
"type": "json_schema",
"json_schema": {
"name": "order",
"strict": True,
"schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"total_cents": {"type": "integer"},
"status": {"type": "string", "enum": ["pending", "paid", "refunded"]}
},
"required": ["order_id", "total_cents", "status"],
"additionalProperties": false
}
}
}
)
print(resp.choices[0].message.content)If you need to pin a provider, pass a routing directive header. Most gateways honor client routing directives; check your docs.
curl https://api.n4n.ai/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-4o",
"messages": [{"role":"user","content":"Return {\"ok\":true}"}],
"response_format": {"type":"json_object"}
}'4. Validate and repair on the client
Structured outputs fail closed when the model refuses or the provider glitches. Always validate the returned string with a real parser, not just json.loads.
from pydantic import BaseModel, ValidationError
class Order(BaseModel):
order_id: str
total_cents: int
status: str
raw = resp.choices[0].message.content
try:
order = Order.model_validate_json(raw)
except ValidationError as e:
# log, fallback to manual parse, or re-ask with corrected schema
print("schema drift:", e)Common pitfall: models return total_cents as a string "1999". Strict schema says integer; some backends coerce, some don’t. Validate types explicitly and cast if your gateway doesn’t.
5. Handle fallback and degradation
A unified gateway earns its keep when a provider rate-limits you. Automatic fallback flips to a secondary model that supports the same schema—if it does. Not all models handle strict: true equally.
Design a degradation path:
- Primary call with strict structured outputs.
- On
422orinvalid_schemaerror, retry with JSON mode and client-side validation. - On persistent failure, return a typed error to the caller.
def get_order(client, text, attempt=0):
try:
return client.chat.completions.create(
model="openai/gpt-4o",
messages=[{"role":"user","content":text}],
response_format=STRICT_SCHEMA
)
except Exception as e:
if attempt == 0:
return client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role":"user","content":text}],
response_format={"type":"json_object"}
)
raiseThe gateway’s automatic fallback may already do this, but client logic avoids silent schema loosening.
6. Cache and meter per token
Structured prompts are repetitive; cache them. Gateways forward provider cache-control hints when you set cache_control in the message or header. Per-token usage metering lets you attribute cost to each schema call.
resp = client.chat.completions.create(
model="openai/gpt-4o",
messages=[
{"role":"system","content":"You emit order JSON.","cache_control":{"type":"ephemeral"}},
{"role":"user","content":"Order #42 $19.99 paid"}
],
response_format=STRICT_SCHEMA
)
print(resp.usage.model_dump()) # prompt_tokens, completion_tokens, cached_tokensTradeoff: caching reduces cost but binds you to one provider’s cache window. If the gateway routes to a different backend on fallback, the cache miss hits your bill.
7. Test against every model you route to
A schema that works on GPT-4o may break on a smaller model. Write a smoke test that loops your model list and asserts the parsed object passes pydantic.
MODELS = ["openai/gpt-4o", "anthropic/claude-3.5-sonnet", "meta/llama-3.1-70b"]
for m in MODELS:
r = client.chat.completions.create(model=m, messages=MSG, response_format=STRICT_SCHEMA)
assert Order.model_validate_json(r.choices[0].message.content)Run this in CI with a capped spend. You’ll catch enum drift and integer coercion issues before users do.
8. Know when not to use structured outputs
If the task is creative or the shape is unbounded (e.g., “list all thoughts”), forced schema strangles the model. Use JSON mode or plain text and parse loosely. The structured outputs unified gateway is a constraint tool, not a default.
Build the schema, route through one endpoint, validate hard, and keep a fallback. That’s the whole job.