JSON schema constraints: a guide for agent builders

Applying JSON schema constraints LLM outputs shifts the contract from best-effort text to machine-checked structure. For agents that trigger tools or persist state, that shift removes an entire class of parsing failures. This guide walks a concrete order of operations for defining, enforcing, and validating those constraints in production systems.

Enforce schema at generation, not in the parser

Post-hoc extraction with json.loads and hope is not a strategy. If the model emits a string instead of an object, or nests a field you expected at the top level, your agent either crashes or silently misbehaves. Native structured output—where the inference server binds the schema during decoding—converts that risk into a provider-side guarantee (where supported).

OpenAI-compatible APIs expose this via response_format of type json_schema. The model is constrained by grammar or post-selection filtering so the returned content matches your schema. Use it whenever the output drives program logic.

from openai import OpenAI

client = OpenAI()  # or any OpenAI-compatible gateway
schema = {
    "type": "object",
    "properties": {
        "action": {"type": "string", "enum": ["search", "reply", "escalate"]},
        "query": {"type": "string"}
    },
    "required": ["action"],
    "additionalProperties": False
}

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Find docs on rate limits"}],
    response_format={"type": "json_schema", "json_schema": {"name": "agent_action", "schema": schema}}
)
print(resp.choices[0].message.content)

Define the smallest schema that forces correctness

Engineers overspecify. A 200-line schema with deep nesting trains the model to hallucinate compliant filler. Start with the minimal set of fields that make the next step in your agent deterministic.

If your agent only needs to decide an action and optionally provide a search string, the three-field schema above is enough. Add additionalProperties: false to reject stray keys. If you need a confidence score later, add it as optional and monitor whether the model uses it.

A common mistake is requiring fields that the model cannot reliably produce. For example, demanding a UUID from the model will cause retries. Generate IDs in your code, not in the prompt.

Model agent state with tagged unions, not oneOf

Complex agents cycle through states: call tool, observe, answer. JSON Schema oneOf is poorly supported by many inference stacks and confuses sampling. Use a single flat object with a discriminant enum and validate the conditional rules in code.

{
  "type": "object",
  "properties": {
    "step": {"type": "string", "enum": ["tool_call", "final_answer"]},
    "tool": {"type": "string"},
    "args": {"type": "object"},
    "answer": {"type": "string"}
  },
  "required": ["step"],
  "additionalProperties": false
}

After parsing, enforce that tool and args are present when step == "tool_call". This keeps the generation constraint simple while preserving logic in your validator.

from pydantic import BaseModel, field_validator, ValidationError

class AgentStep(BaseModel):
    step: str
    tool: str | None = None
    args: dict | None = None
    answer: str | None = None

    @field_validator("tool", "args")
    def required_for_tool_call(cls, v, info):
        if info.data.get("step") == "tool_call" and v is None:
            raise ValueError("tool and args required for tool_call")
        return v

Validate at the boundary with a real library

Never trust the provider completely. Some implementations loosen additionalProperties or mangle number formats. Run the raw string through a validator the moment it enters your system.

import json
from pydantic import ValidationError

raw = resp.choices[0].message.content
try:
    data = AgentStep.model_validate(json.loads(raw))
except (json.JSONDecodeError, ValidationError) as e:
    # route to fallback or ask model to retry with correction hint
    log_and_recover(e)

If you need schema-versioned evolution, append a schema_version field and branch validation. Agents break when schemas change silently.

Streaming trades latency for validation pain

Streaming token output is incompatible with strict schema enforcement unless the server supports streaming structured outputs (few do). If you stream, buffer the full payload, then validate. Do not attempt to act on partial JSON—an agent that fires a tool call on a half-parsed object will corrupt state.

Where low latency matters more than guaranteed structure, use plain JSON mode (response_format: {"type": "json_object"}) and accept that you must validate. JSON mode only promises syntactically valid JSON, not conformance to your JSON schema constraints LLM prompt. That distinction causes production incidents.

Route across providers without rewriting validation

Multi-model agents hit provider rate limits and regional degradation. If you hardcode schema handling per vendor, you multiply surface area. An OpenAI-compatible gateway that honors the same response_format across models lets you keep one validation path.

n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and honors client routing directives, so the identical JSON schema constraints LLM request can fail over when a provider is rate-limited without changing your parser. The fallback preserves the schema contract; your Pydantic model stays the single source of truth.

When you do route, forward provider cache-control hints if your gateway supports them. Schema-heavy prompts benefit from prefix caching; a missed cache hint doubles latency and cost.

Common pitfalls and tradeoffs

Over-constraining kills task performance. A schema that demands exact field names for free-form user intent forces the model to map speech to your ontology. Keep enums small and let query or description carry the rest.

Enum drift. When you add an action, old agents may not know it exists. Version the schema and log which enum values actually appear. Dead enum entries confuse the model.

Nested arrays of objects blow up context. A schema allowing items: {type: array, items: {…}} with depth three can consume thousands of tokens in the grammar alone. Flatten or paginate.

JSON mode vs structured outputs. JSON mode is supported widely but ignores your shape. Structured outputs (json_schema) are stricter but not available on every open-weight model. Know which you call.

Repair vs reject. Sometimes the model returns {"action": "search", "query": null} when you needed a string. A lightweight repair step—coercing null to empty string—keeps the agent alive. But never silently mutate semantics; log the repair.

Ordered path for agent builders

1. Write the minimal schema that makes the next agent step deterministic. Disable additionalProperties.
2. Attach it via response_format of type json_schema on an OpenAI-compatible client.
3. Parse with a strict validator (pydantic/jsonschema) at the system boundary.
4. Apply conditional rules the schema can’t express (tagged union checks).
5. If streaming, buffer then validate; never act on partial JSON.
6. Route across models through a single compatible endpoint to avoid schema fragmentation.
7. Version schemas, log enum usage, and review repairs weekly.

Following this order turns JSON schema constraints LLM outputs from a hope into a tested interface. Your agent gains predictability without sacrificing the model’s reasoning flexibility.