When a primary model provider goes down, your production chat breaks unless you’ve implemented automatic failover for LLM outages. This tutorial walks through building a minimal but production-minded failover client that swaps to a backup provider on errors, with health checks and circuit breaking. You’ll leave with runnable Python that you can drop into a service.
Prerequisites
- Python 3.11 or newer
openaiandanthropicPython packages (pip install openai anthropic)- API credentials for at least two providers exported as
OPENAI_API_KEYandANTHROPIC_API_KEY - Comfort with synchronous code; the same shape applies to
asyncvariants
Step 1: Define a uniform provider interface
Providers expose different SDK shapes. Wrap them behind a tiny protocol so the failover logic never cares about the underlying client.
from typing import Protocol
class ChatProvider(Protocol):
def complete(self, prompt: str, model: str | None = None) -> str:
...Now concrete adapters. These are thin and set sane timeouts.
import os
from openai import OpenAI
from anthropic import Anthropic
class OpenAIAdapter:
def __init__(self, model: str = "gpt-4o-mini"):
self.client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
self.model = model
def complete(self, prompt: str, model: str | None = None) -> str:
resp = self.client.chat.completions.create(
model=model or self.model,
messages=[{"role": "user", "content": prompt}],
timeout=10,
)
return resp.choices[0].message.content
class AnthropicAdapter:
def __init__(self, model: str = "claude-3-5-sonnet-20240620"):
self.client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
self.model = model
def complete(self, prompt: str, model: str | None = None) -> str:
resp = self.client.messages.create(
model=model or self.model,
max_tokens=512,
messages=[{"role": "user", "content": prompt}],
)
return resp.content[0].textStep 2: Naive fallback chain
The simplest version of automatic failover for LLM outages is an ordered list of providers and a try/except loop.
class FailoverClient:
def __init__(self, providers: list[ChatProvider]):
self.providers = providers
def complete(self, prompt: str, model: str | None = None) -> str:
last_err = None
for p in self.providers:
try:
return p.complete(prompt, model)
except Exception as e:
last_err = e
print(f"provider {p.__class__.__name__} failed: {e}")
raise RuntimeError("all providers failed") from last_errRun it:
if __name__ == "__main__":
fo = FailoverClient([OpenAIAdapter(), AnthropicAdapter()])
print(fo.complete("Say hi in three words"))Expected output when OpenAI is healthy:
Hi there! How's it?If OpenAI is timing out, you’ll see:
provider OpenAIAdapter failed: Request timed out.
Hi! How can I help?That works, but it fails over on any error—including a 400 from malformed input, which a backup will also reject.
Step 3: Classify transient errors
Only connection failures, timeouts, and 5xx-style degradation should trigger failover. Both SDKs expose specific exception types.
from openai import APIError, APITimeoutError
from anthropic import APIError as AnthropicAPIError
def is_transient(e: Exception) -> bool:
if isinstance(e, (APITimeoutError, AnthropicAPIError, APIError)):
# OpenAI APIError has a status_code attribute
status = getattr(e, "status_code", None)
if status is None or status >= 500:
return True
return False
# catch-all for connection errors
return "connection" in str(e).lower() or "timeout" in str(e).lower()Thread is_transient into the loop: re-raise on non-transient errors so you don’t burn a backup on a bad request.
Step 4: Add a circuit breaker
Once a provider starts failing, hammering it amplifies the outage. A circuit breaker marks a provider dead after N consecutive transient failures and skips it for a cooldown window.
import time
class CircuitBreaker:
def __init__(self, threshold: int = 3, cooldown: int = 30):
self.fails = 0
self.threshold = threshold
self.cooldown = cooldown
self.opened_at = 0.0
def allow(self) -> bool:
if self.fails >= self.threshold:
if time.time() - self.opened_at > self.cooldown:
self.fails = 0
return True
return False
return True
def record_success(self) -> None:
self.fails = 0
def record_failure(self) -> None:
self.fails += 1
if self.fails == self.threshold:
self.opened_at = time.time()Wire it into the client:
class FailoverClient:
def __init__(self, providers: list[ChatProvider]):
self.providers = [(p, CircuitBreaker()) for p in providers]
def complete(self, prompt: str, model: str | None = None) -> str:
last_err = None
for p, cb in self.providers:
if not cb.allow():
print(f"skipping {p.__class__.__name__}: circuit open")
continue
try:
res = p.complete(prompt, model)
cb.record_success()
return res
except Exception as e:
if not is_transient(e):
raise
cb.record_failure()
last_err = e
print(f"provider {p.__class__.__name__} transient fail: {e}")
raise RuntimeError("all providers failed") from last_errAfter three strikes, the loop logs skipping OpenAIAdapter: circuit open and goes straight to Anthropic.
Step 5: Proactive health polling
Reacting to errors is necessary, but you can also demote a provider before a user hits it. Add a health() method to each adapter that calls a lightweight endpoint.
class OpenAIAdapter:
# ... existing code ...
def health(self) -> bool:
try:
self.client.models.list()
return True
except Exception:
return False
class AnthropicAdapter:
# ... existing code ...
def health(self) -> bool:
try:
self.client.models.list()
return True
except Exception:
return FalseA background thread or cron can call health() and pre-open circuits. In practice, the error-driven breaker from Step 4 is enough for most teams; poll only if you have tight SLOs.
Step 6: Simulate an outage and verify
Create a broken adapter to stand in for a provider outage:
from openai import APITimeoutError
class BrokenAdapter(OpenAIAdapter):
def complete(self, prompt: str, model: str | None = None) -> str:
raise APITimeoutError("simulated outage")
if __name__ == "__main__":
fo = FailoverClient([BrokenAdapter(), AnthropicAdapter()])
for i in range(5):
try:
print(f"attempt {i}: {fo.complete('ping')[:20]}")
except Exception as e:
print(f"attempt {i}: all failed ({e})")Expected output:
provider BrokenAdapter transient fail: simulated outage
attempt 0: Hi! How can I help
provider BrokenAdapter transient fail: simulated outage
attempt 1: Hi! How can I help
provider BrokenAdapter transient fail: simulated outage
skipping BrokenAdapter: circuit open
attempt 2: Hi! How can I help
skipping BrokenAdapter: circuit open
attempt 3: Hi! How can I help
skipping BrokenAdapter: circuit open
attempt 4: Hi! How can I helpThe circuit opens after three failures and the client stops calling the dead provider, protecting both your latency budget and the provider’s recovery.
Closing notes
Building automatic failover for LLM outages is mostly disciplined error classification plus a little stateful health tracking—not exotic infrastructure. The pattern above scales to three or more providers, and you can swap the adapters for any OpenAI-compatible endpoint.
If you’d rather not operate this logic yourself, a gateway like n4n.ai provides automatic fallback when a provider is rate-limited or degraded across 240+ models behind one OpenAI-compatible endpoint, including per-token metering. But rolling your own is perfectly viable if you keep the transient-error boundary strict and the circuit breaker tuned to your traffic shape.