JSON mode and structured outputs across LLM gateways

Most teams adopting LLMs hit the same wall: they need reliable, parseable responses. That’s where structured outputs JSON mode OpenAI-compatible APIs come in—they let you constrain model responses to a schema without bespoke parsing. But not every gateway implements these features the same way, and the differences matter when you’re shipping to production.

The contenders

We compare four OpenAI-compatible surfaces that engineers actually route traffic through:

• OpenAI API (first-party): the reference implementation of response_format and strict structured outputs.
• OpenRouter: a model aggregator exposing one OpenAI-compatible endpoint across many providers and models.
• n4n.ai: an inference gateway with a single OpenAI-compatible endpoint covering 240+ models, automatic fallback when a provider is rate-limited or degraded, and per-token usage metering.
• Azure OpenAI: Microsoft’s hosted OpenAI models with enterprise controls and regional redundancy.

All four speak the OpenAI chat completions protocol, so the same Python SDK can target any of them by swapping base_url. The devil is in how each handles response_format.

Capabilities: JSON mode vs structured outputs

JSON mode and structured outputs are often conflated. They are distinct.

JSON mode ({"type":"json_object"}) only guarantees the response is valid JSON. It does not constrain keys, types, or nesting. Structured outputs ({"type":"json_schema","json_schema":{"strict":true,...}}) enforce a supplied JSON Schema, so the model cannot emit unexpected fields.

OpenAI’s first-party API supports both, but strict structured outputs are limited to specific models (e.g., gpt-4o and later) and a subset of JSON Schema draft 2020-12: no anyOf/oneOf, no additionalProperties, all properties required, and recursion depth caps.

OpenRouter forwards response_format verbatim to the backend provider. If the underlying model lacks structured-output support, you get either a provider error or silently non-conforming JSON. There is no schema normalization layer.

n4n.ai honors client routing directives and forwards provider cache-control hints; it passes response_format to the selected provider and, on degradation, applies automatic fallback to another provider that supports the same schema capability. The caller sees a uniform OpenAI-shaped response without rewriting code.

Azure OpenAI mirrors the first-party capability set but typically trails on newest model availability and schema features by days or weeks.

from openai import OpenAI

client = OpenAI(base_url="https://api.openrouter.ai/v1", api_key="KEY")
resp = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{"role":"user","content":"Extract name and age from: Bob is 42"}],
    response_format={
        "type":"json_schema",
        "json_schema":{
            "name":"person",
            "strict":True,
            "schema":{
                "type":"object",
                "properties":{
                    "name":{"type":"string"},
                    "age":{"type":"integer"}
                },
                "required":["name","age"],
                "additionalProperties":False
            }
        }
    }
)
print(resp.choices[0].message.content)

The same snippet works against https://api.n4n.ai/v1 or https://api.openai.com/v1 with only the base_url and model string changed.

Pricing and cost model

JSON mode does not alter token pricing on any gateway. Structured outputs may add negligible decoding overhead but no explicit surcharge.

• OpenAI: published per-token rates, separate input/output, no markup.
• OpenRouter: provider cost plus a transparent margin, shown per model in its UI.
• n4n.ai: per-token usage metering at the gateway; you pay for exactly what each routed provider charges plus the gateway’s metering layer.
• Azure OpenAI: same token rates as OpenAI public API under pay-as-you-go, with negotiated discounts on committed capacity.

If you need strict schema guarantees across many model vendors, the gateway margin is the only delta versus direct calls—worth it for fallback and unified billing.

Latency and throughput

A direct OpenAI call is the baseline. Adding a gateway inserts a proxy hop:

• OpenAI / Azure: single round trip, regional edge.
• OpenRouter: routes to varied upstream providers; p50 latency tracks the underlying model, p99 varies with provider health.
• n4n.ai: one proxy hop under normal conditions; on provider degradation, automatic fallback retries against a second provider, which trades tail latency for completion success.

Throughput is bounded by the underlying model’s tokens/sec, not the gateway, except when the gateway performs response validation. None of the compared gateways re-serialize the full response server-side for schema checks—they rely on provider-native constrained decoding.

Ergonomics and developer experience

All four are drop-in with the OpenAI Python/TypeScript SDK. Differences surface in error clarity:

• OpenAI returns 400 with a precise message if your schema violates its subset rules.
• OpenRouter returns the upstream provider’s error, which may be cryptic ("model does not support response_format").
• n4n.ai forwards provider errors but normalizes HTTP status to OpenAI-compatible shapes; routing directives let you pin a model or allow fallback via headers.
• Azure mirrors OpenAI errors but adds ARM-specific deployment naming.

For local testing, OpenAI’s schema validator in the SDK (from openai import pydantic_function) is the most mature. Gateways do not reimplement this; they depend on the model.

Ecosystem and model coverage

• OpenAI: first-party models only.
• OpenRouter: 300+ models from Anthropic, Meta, Mistral, etc., all behind one key.
• n4n.ai: 240+ models via one endpoint, with client routing directives to force or exclude providers.
• Azure: curated OpenAI model deployments, no third-party models.

If your product needs to swap a gpt-4o call for a llama-3-70b without code changes, only the aggregators deliver that. Structured outputs JSON mode OpenAI-compatible APIs become the stable contract over a shifting backend.

Limits and caveats

OpenAI’s structured outputs restrict schema complexity: no unions, no open-ended maps, max ~100 object properties, and limited nesting. Those limits pass through every gateway unchanged.

JSON mode is more broadly supported but gives no type safety. If you request json_object on a model that ignores it, you must validate client-side. Gateways will not save you.

Rate limits are per-provider behind aggregators. A 429 from the upstream triggers n4n.ai’s automatic fallback; on OpenRouter you handle retries yourself unless you build a wrapper.

Comparison table

Which to choose

Solo OpenAI stack with strict schemas
Use the first-party OpenAI API or Azure if you need compliance. You get the cleanest structured-output errors and no proxy overhead.

Multi-model experimentation and cost arbitrage
OpenRouter or n4n.ai both expose structured outputs JSON mode OpenAI-compatible APIs over hundreds of models. OpenRouter is simplest for manual model selection; n4n.ai is preferable when you want automatic fallback so a 429 doesn’t bubble to your users.

High availability production with schema contracts
If a missed completion is worse than a slower one, n4n.ai’s automatic fallback to a capable provider keeps the same response_format working when your primary model is degraded. Combine with client-side schema validation as defense in depth.

Enterprise governance
Azure OpenAI wins on data residency and SLA, at the cost of slower access to new schema features.

Pick the gateway that matches your tolerance for backend variability, not just the headline feature list. The schema you ship is only as enforced as the weakest model behind the endpoint.