Client-side routing models array vs provider object describes two distinct ways an API client tells an LLM gateway which backends to try for a request. The models array is an ordered list of fully-qualified model identifiers; the provider object is a structured routing directive that controls provider order, fallbacks, and cache hints. Both shift routing decisions from the gateway’s default policy to the caller, but they differ in precision and overhead.
What client-side routing means
In an LLM inference gateway, routing decides which upstream provider and model serves a given request. Server-side routing uses the gateway’s own health checks, cost rules, and round-robin logic. Client-side routing pushes those preferences into the request body: the caller declares what it will accept and in what order.
This matters because the client understands its own latency budget, compliance constraints, and quality bar. A payments bot may refuse to hit a smaller open-weight model; a bulk summarizer may happily fall back to anything cheap. Encoding that in the request avoids round trips and misrouting.
The phrase client-side routing models array vs provider object is really about two serialization formats for those preferences. One is flat and model-centric. The other is nested and provider-centric.
The models array: syntax and semantics
The models array puts a JSON array where you’d normally put a model string. Each entry is a fully-qualified model ID, typically provider/model-name.
{
"model": ["openai/gpt-4o", "anthropic/claude-3.5-sonnet", "meta-llama/llama-3.1-70b"],
"messages": [{ "role": "user", "content": "Extract entities" }]
}The gateway attempts the first entry. If that provider returns a 429, 5xx, or a timeout, it moves to the second, then the third. Success on any entry returns normally; the client never sees the failed attempts unless it inspects response headers.
Key properties:
- Ordered: position implies priority.
- Opaque to provider details: you specify model IDs, not the underlying provider account or region.
- Coarse: you cannot say “try OpenAI in Azure first, then OpenAI direct” because those are distinct provider entries only if the model ID encodes them (e.g.,
azure/gpt-4ovsopenai/gpt-4o).
The provider object: syntax and semantics
The provider object separates the model name from routing constraints. It is a nested object, often under a provider key, that tells the gateway how to select among backends that can serve the named model.
{
"model": "gpt-4o",
"provider": {
"order": ["OpenAI", "Azure", "Bedrock"],
"allow_fallbacks": true,
"cache_control": { "type": "ephemeral" }
},
"messages": [{ "role": "user", "content": "Draft a contract" }]
}Here the client asks for gpt-4o but constrains the gateway to try the OpenAI direct endpoint first, then Azure’s deployment, then Bedrock’s. The allow_fallbacks flag permits the gateway to substitute a different model only if the provider object permits it (some gateways treat this as “fall back to same model on other providers” vs “fall back to any model”). The cache_control field forwards caching hints to the provider.
Key properties:
- Provider-aware: you address routing at the vendor level, not just the model string.
- Extensible: can carry cache directives, region pins, or data-residency flags.
- Model-agnostic at the top level: the
modelfield stays a single string.
client-side routing models array vs provider object: key differences
The distinction is not just cosmetic.
- Granularity: the models array lists concrete model endpoints; the provider object lists vendors that may each serve the same model.
- Fallback scope: with an array, fallback implies a different model (or same model on a different provider if IDs differ). With a provider object, fallback stays within one model across providers unless explicitly widened.
- Cache and metadata: only the provider object routinely carries cache-control or compliance metadata. The array is pure target selection.
- Composability: some gateways let you combine both—an array of model strings where each entry can have an attached provider object—but that is rare and verbose.
When evaluating client-side routing models array vs provider object, treat the array as a quick fallback list and the object as a policy fragment.
Why it matters for failover and fallback
Automatic failover is the headline use case. Providers degrade. A region goes yellow, a quota exhausts, a model deprecates. If the client hardcodes a single model string, the gateway must either return an error or apply its own fallback policy that the client may not trust.
By sending routing directives, the client bounds the blast radius. With the models array, a 429 from OpenAI becomes a silent retry against Anthropic. With the provider object, a 429 from OpenAI direct becomes a retry against Azure before any cross-model jump.
A gateway like n4n.ai honors client routing directives and forwards provider cache-control hints, so the provider object’s cache field survives the trip to the upstream rather than being stripped at the edge. That preserves prompt caching across fallback attempts.
Without client-side routing, you force the gateway to guess. Guessing causes latency spikes when it tries disallowed models, or compliance violations when it hits a provider the client vetoed.
How gateways implement the fallback
When a request arrives with a models array, the gateway loops internally. It rewrites the outbound request to the first model, calls the provider, and inspects the result. On a transport error or explicit 429/5xx, it advances the index. The client receives a single HTTP response; the fallback is invisible except via a response header like x-routed-model.
For the provider object, the gateway resolves the model to a set of provider endpoints, sorts by order, and applies the same loop. If allow_fallbacks is false and the first provider fails, it returns the error immediately.
This internal loop means client-side routing does not cost extra round trips for the client. The latency penalty is borne by the gateway’s outbound calls.
Concrete example with code
Assume an OpenAI-compatible endpoint. The Python client is unchanged; only the model field or extra body differs.
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.example/v1",
api_key="sk-...",
)
# Models array style: try GPT-4o, then Claude, then Llama
resp = client.chat.completions.create(
model=["openai/gpt-4o", "anthropic/claude-3.5-sonnet", "meta-llama/llama-3.1-70b"],
messages=[{"role": "user", "content": "Summarize the incident"}],
)
print(resp.model) # shows which actually served# Provider object style: stay on gpt-4o, but prefer Azure over direct
resp2 = client.chat.completions.create(
model="gpt-4o",
extra_body={
"provider": {
"order": ["Azure", "OpenAI"],
"allow_fallbacks": True,
"cache_control": {"type": "ephemeral"}
}
},
messages=[{"role": "user", "content": "Summarize the incident"}],
)In the first call, if OpenAI is down, the gateway shifts to Claude without client involvement. In the second, the gateway keeps the same model family but switches the vendor account, preserving output format expectations.
Streaming and partial failures
Streaming complicates fallback. Once the first token streams, the gateway has committed to a backend. If that backend dies mid-stream, the SSE connection errors; the gateway cannot silently switch because the client already received content. Therefore fallback applies only to the pre-first-token window.
Engineers often assume the models array protects streaming too. It does not beyond connection establishment. Design clients to handle stream errors and retry the whole request if idempotency allows.
Common misconceptions
“The models array means load balancing.” No. It is an ordered fallback list, not a round-robin pool. The gateway tries entry zero every time unless it fails. If you want distribution, you need server-side policy or randomize the array client-side (which defeats determinism).
“The provider object overrides all gateway logic.” Incorrect. The gateway still applies its own health checks. If you list Bedrock first but Bedrock is unreachable from the gateway’s region, the gateway may skip it faster than your order implies. The object is a hint, not a contract.
“You must pick one format.” Not true. Some requests benefit from both: an array of model strings, each with a nested provider object for fine control. But that complexity is justified only in regulated pipelines.
“Fallback is free.” Fallback adds latency on the unhappy path and may double-count tokens if the first provider partially processed a long prompt before erroring. Client-side routing does not eliminate that cost; it only makes the attempt explicit.
“Cache hints always work.” Cache-control fields are forwarded only if the gateway and provider both support them. A provider object declaring ephemeral caching on a model that doesn’t support it is ignored upstream.
Practical recommendations
Use the models array when:
- You genuinely accept different model families as equivalent.
- You want minimal request size and maximum portability across gateways.
- Your fallback need is “anything that returns a token.”
Use the provider object when:
- You must stay on one model for output schema stability.
- You have compliance or latency reasons to prefer a specific vendor deployment.
- You need to forward cache-control or residency flags.
If you are building a library on top of a gateway, expose both. Let the application developer choose. Hide the serialization behind a Route builder so the difference between client-side routing models array vs provider object is a constructor argument, not a JSON hack.
Finally, test fallback explicitly. Force a 429 in staging by pointing the first array entry at a dead model ID. Verify the second entry serves and that your logs record which backend won. Routing code that has never seen a failure is routing code you do not understand.