Most LLM integrations assume the model endpoint will be there. Building resilient LLM apps automatic failover means planning for rate limits, regional outages, and silent degradations before they page you at 3am.
1. Map the failure modes you actually hit
Providers don’t fail uniformly. If you treat every error as a generic “retry,” you’ll mask the real problem and burn money. Break failures into categories you can observe and act on.
Rate limits and quotas
The 429 is the most common. It can be per-key, per-org, or per-model. Some providers return retry-after; others don’t. Treat a missing header as “back off exponentially.”
Hard outages
A 500/502/503 or a connection timeout. These usually clear in seconds, but regional DNS issues can last minutes.
Silent degradation
The request succeeds, but the model returns truncated JSON, ignores the system prompt, or stalls mid-stream. This is the nastiest class because your failover logic won’t trigger.
Behavior drift
Even with the same model name, two providers (or two versions) format tool calls differently. Your parser breaks, not the API.
Log the error class per request. Resilient LLM apps automatic failover starts with knowing which bucket dominates your traffic.
2. Isolate the model call behind a thin client
Scattering openai SDK calls across controllers makes failover impossible. Put every completion behind one function that takes a model slug and returns a normalized result.
from openai import OpenAI, APIError, RateLimitError
import os, time
def complete(prompt: str, model: str, base_url: str, api_key: str, timeout=10) -> str:
client = OpenAI(base_url=base_url, api_key=api_key)
start = time.monotonic()
try:
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
timeout=timeout,
)
return resp.choices[0].message.content
except (RateLimitError, APIError) as e:
# attach context for caller
e.provider = base_url
raiseSet a hard timeout. A default 60s client timeout turns a provider hiccup into a user-visible hang. Fail fast, fail upward.
3. Choose a failover strategy: redirect, retry, or degrade
You have three levers. Pick based on the failure class.
Same model, secondary provider
If you run GPT-4o on both OpenAI and Azure, a 429 on one can be served by the other. This preserves quality.
def complete_with_failover(prompt, model, primary, secondary, fallback_model=None):
try:
return complete(prompt, model, primary["url"], primary["key"])
except RateLimitError:
try:
return complete(prompt, model, secondary["url"], secondary["key"])
except RateLimitError:
if fallback_model:
return complete(prompt, fallback_model, primary["url"], primary["key"])
raiseDowngrade to a smaller model
When capacity is gone, a 4o-mini with a stricter prompt beats a 503. Accept lower quality for availability.
Degrade to cache or template
For low-stakes queries, return a cached answer from a similar prompt. Use embedding similarity to find it.
Pitfall: sequential failover multiplies tail latency. If primary takes 9s to timeout, and secondary takes 9s, your p99 becomes 18s. Use a circuit breaker to mark a provider dead for 30s after N failures.
If you’d rather not hand-roll this, a gateway like n4n.ai collapses the boilerplate: one OpenAI-compatible endpoint that addresses 240+ models with automatic fallback when a provider is rate-limited or degraded. It honors client routing directives and forwards provider cache-control hints, so your fallback stays transparent.
Resilient LLM apps automatic failover isn’t just retry loops—it’s a deliberate routing policy with latency budgets.
4. Make requests idempotent and safe to replay
Failover replays requests. If your LLM call triggers a side effect (send email, write DB row, call webhook), you must separate generation from action.
import uuid, kv
def handle_query(req):
rid = req.id or str(uuid.uuid4())
cached = kv.get(f"llm:{rid}")
if cached:
return cached
text = complete_with_failover(req.prompt, "gpt-4o", PRIMARY, SECONDARY)
kv.set(f"llm:{rid}", text, ttl=3600)
return textFor streaming, mid-stream drops are not replayable without checkpointing. Buffer the full response server-side, then stream from your own store. Otherwise a failover mid-stream corrupts the client’s view.
Tradeoff: buffering adds a small latency tax but gives you exactly-once generation semantics.
5. Instrument everything per-token
You can’t tune what you don’t measure. Emit per-attempt metrics: provider, model, prompt_tokens, completion_tokens, latency_ms, and whether it was a fallback attempt.
log.info("llm_call",
model=model,
provider=base_url,
prompt_tokens=resp.usage.prompt_tokens,
completion_tokens=resp.usage.completion_tokens,
latency_ms=int((time.monotonic()-start)*1000),
fallback=was_fallback)Alert on failover rate, not just error rate. A 5% fallback rate means your primary is chronically constrained. Per-token metering also exposes the real cost of a parallel race: you may pay for two completions when both return.
Observability is the backbone of resilient LLM apps automatic failover. Without per-token splits, you’ll discover cost spikes in the invoice, not the dashboard.
6. Cache aggressively to reduce failover surface
Every cached hit is a request that never touches a provider. Use a semantic cache keyed on embedding similarity with a threshold of ~0.95.
def cached_complete(prompt, model):
emb = embed(prompt)
hit = vector_db.search(emb, threshold=0.95)
if hit:
return hit.text
return complete_with_failover(prompt, model, PRIMARY, SECONDARY)This cuts spend and removes a whole class of failover events. Invalidate on model version changes.
7. Test failover like you mean it
Write chaos tests that force the primary to fail in the ways you mapped in step 1.
# primary points to a black hole
export PRIMARY_URL="http://localhost:9999"
python -c "from app import cached_complete; print(cached_complete('summarize this','gpt-4o')[:50])"Use Toxiproxy to inject latency, not just refused connections. Real failover pain comes from a provider that accepts the TCP connection but hangs for 30s.
Run a game day: kill the primary provider’s API key, watch dashboards, confirm fallbacks fire and latencies stay under budget.
Tradeoffs you should accept upfront
- Consistency: Same model name on two providers can still differ in system prompt handling or tool-call schema. Test both.
- Cost: Failover doubles spend on raced requests. Budget for it.
- Complexity: Hand-rolled logic is ongoing maintenance. A gateway is an external dependency but removes code.
- Latency: Every fallback hop adds worst-case seconds. Set a hard latency budget and degrade instead of retrying past it.
Pick the strategy per endpoint. A user-facing chat can degrade to a smaller model; a batch summarization job can retry for minutes. Resilient LLM apps automatic failover is a per-workload decision, not a global flag.