A provider’s 99.9% SLA does not translate to 99.9% usable availability when rate limits, degraded inference, or regional outages hit your specific traffic pattern. Building automatic failover LLM uptime into your request path is the only practical way to push effective availability past the weakest link in your model supply chain.
The gap between contractual and effective uptime
Cloud LLM providers publish SLAs that cover control-plane reachability, not task success. A 200 OK with a truncated completion or a 429 after ten seconds both count against your users but may not count against the provider’s numbers. Effective uptime is the percentage of your requests that return a usable response within your latency budget.
I have watched a “healthy” provider return 503s for one model region while its status page stayed green. If your code only talks to that endpoint, your feature is down even though the SLA is intact. Automatic failover LLM uptime treats provider health as a per-request property, not a dashboard color.
Measure what your users feel
Define a success criterion before writing routing code:
- Response arrived before
max_latency_ms. - Completion contains expected structure (JSON, tool call, or non-empty text).
- No intermediate auth or quota error.
Instrument each attempt. A simple counter per provider and per model tells you where time goes. Per-token usage metering helps you spot a fallback that silently doubles cost.
from dataclasses import dataclass
@dataclass
class AttemptResult:
provider: str
model: str
ok: bool
latency_ms: int
tokens: intLog these. Do not aggregate away the tails. If your p99 latency budget is 4 seconds and a fallback adds 1.5 seconds of retry overhead, you have converted a hard error into a slow failure that looks like uptime but feels like outage.
Compute effective uptime as successful_completions / total_requests over a rolling window. Run this number alongside provider SLAs in your weekly review. The delta is why you built failover in the first place.
Build automatic failover LLM uptime: an ordered path
Follow this sequence. Skip a step and you will debug ghost outages at 3 a.m.
Step 1: Unify the interface
Standardize on an OpenAI-compatible request shape. Most gateways and several open-weight servers accept the same /v1/chat/completions contract. This lets you swap base URLs without rewriting prompt logic.
from openai import OpenAI
client = OpenAI(
base_url="https://api.n4n.ai/v1", # or your own proxy
api_key="sk-...",
)If you later add a self-hosted vLLM node, the call site stays identical. The only change is the base_url and the model string.
Step 2: Detect failures precisely
Catch only what is retryable. A 400 from malformed JSON is your bug, not a failover trigger. A 429, 500, 503, or socket timeout is.
from openai import APIError, RateLimitError, APITimeoutError
class TransientFailure(Exception):
pass
def call_with_catch(client, **kwargs):
try:
return client.chat.completions.create(**kwargs)
except (RateLimitError, APITimeoutError, APIError) as e:
if getattr(e, "status_code", 0) in (429, 500, 503):
raise TransientFailure(e)
raiseDo not treat a 400 as transient. I have seen teams loop on a bad schema and burn quota across three providers.
Step 3: Define fallback precedence
Order candidates by capability, then cost. Do not fall back to a tiny model that cannot emit the schema your parser expects. Encode tiers in config.
{
"routes": [
{"provider": "alpha", "model": "big-2024", "max_cost_per_1k": 0.01},
{"provider": "beta", "model": "mid-2024", "max_cost_per_1k": 0.004},
{"provider": "gamma", "model": "small-2024", "max_cost_per_1k": 0.001}
]
}Automatic failover LLM uptime only works if the fallback model can actually satisfy the request. Test that assumption offline. If your primary model supports function calling and the fallback does not, route around it or degrade gracefully by parsing text.
Honor any client routing directives you expose. If a caller passes x-prefer-provider: beta, your failover loop should try beta first but still escape to gamma on transient error.
Step 4: Preserve context across switches
If you retry after a stream starts, you may have already sent the system prompt and three user turns. Re-sending the full context is fine for stateless calls, but if you used provider prefix caching, a new provider loses that cache. Forward cache-control hints only when the target honors them.
For partial streams, capture emitted tokens and append them as assistant context on retry:
messages = [{"role": "system", "content": sys_prompt}]
messages += history
if partial_assistant:
messages.append({"role": "assistant", "content": partial_assistant})This prevents the model from repeating the half-sentence the user already saw.
Step 5: Meter and alert per route
Track tokens and latency per provider. A fallback that fires on 5% of traffic but costs 3x is a budget leak, not a win. Set alerts on fallback rate, not just error rate.
# example alert rule (Prometheus-ish)
alert: HighFallbackRate
expr: sum(rate(failover_attempts_total[5m])) / sum(rate(llm_requests_total[5m])) > 0.05Per-token usage metering should land in the same dashboard as error rate. If you cannot attribute cost to a route, you cannot tune the list.
Code: minimal client with failover
Below is a compact loop that tries routes in order. It is not production-complete but shows the shape.
def complete(routes, messages, client_factory):
last_err = None
for route in routes:
client = client_factory(route)
try:
resp = call_with_catch(
client,
model=route["model"],
messages=messages,
timeout=8.0,
)
return resp, route
except TransientFailure as e:
last_err = e
continue
raise last_errThe client_factory returns an OpenAI instance pointed at the correct base URL. If you use a gateway that already routes, this loop collapses to a single call.
Test failover without waiting for outages
Fault injection beats hope. Run a local proxy that returns 503 for the primary route 20% of the time, then watch your metrics.
# using a simple mitmproxy script or nginx 503 map
location /v1/chat/completions {
if ($arg_fail = "1") { return 503; }
proxy_pass https://real-provider;
}Drive a canary percentage of traffic through the broken route in staging. Verify the fallback fires, the parser still works, and latency stays under budget. I have found more bugs in fallback logic this way than from real incidents.
Common pitfalls and tradeoffs
Latency tail grows. Each failed attempt adds round-trip time. Cap attempts at two or three. Users prefer a fast error over a 30-second zombie.
Model behavior drift. Fallback models paraphrase differently. If you rely on exact phrase matching downstream, you will ship a silent regression. Validate outputs with the same parser for every route.
Cache misses. Provider-side prefix caches are per-account and per-region. A failover defeats them. n4n.ai forwards provider cache-control hints when the target supports them, but cross-provider caching is impossible. Accept the miss or keep a local KV cache of normalized prompts.
Streaming state. Mid-stream failure leaves partial tokens. Your caller must handle incomplete JSON. Build a resolver that retries only the missing tail, not the whole call.
Cost opacity. Automatic failover LLM uptime can mask a provider that is permanently degraded but cheap. Review per-token spend weekly.
Health signal staleness. If you cache provider health for five minutes, you will keep hitting a dead route. Use short TTLs and per-model granularity; a provider can be fine for one model and dead for another.
When a gateway already does it
Running your own routing mesh is justified at scale, but many teams only need a smart endpoint. A gateway such as n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models and applies automatic fallback when a provider is rate-limited or degraded. That removes the client-side loop above for the common case. You still own conversation-level retry and schema validation, because no gateway knows your success criterion.
If you adopt a gateway, keep your Step 2 detection: the gateway may return a synthetic 200 with an error payload if misconfigured. Trust but verify.
Final checklist
- Defined success beyond HTTP 200.
- Unified interface via OpenAI-compatible schema.
- Distinguished transient from permanent errors.
- Ordered fallbacks by capability and cost.
- Handled partial streams and cache hints.
- Metered tokens per route and alerted on fallback rate.
- Injected faults in staging to prove the path works.
Ship the simplest version that retries once. Expand routing only after data shows which provider actually fails. Effective uptime is an empirical property, not a configuration flag.