Building resilient agentic systems means planning for provider outages before they happen. This multi-provider failover tutorial shows you how to wire automatic fallback into a production LLM agent using nothing but standard OpenAI-compatible calls and a thin client wrapper. You will leave with runnable code that degrades gracefully when a model vendor throws 429s or drops dead.
Prerequisites
- Python 3.11 or newer
openaiPython SDK >= 1.40 (providesOpenAI,RateLimitError,APIError)tenacityif you want retry decorators (optional for this pattern)- API keys for at least two OpenAI-compatible providers. I’ll use OpenAI and Groq, but any endpoint that speaks the
/v1/chat/completionsshape works. - Environment variables:
OPENAI_API_KEY,GROQ_API_KEY.
pip install openai tenacity
export OPENAI_API_KEY=sk-...
export GROQ_API_KEY=gsk-...Step 1: Declare your provider chain
Order matters. Put the provider with the best latency or quality first, and the cheap backup last. The wrapper will try them in sequence and stop at the first success.
import os
PROVIDERS = [
{
"name": "openai",
"base_url": "https://api.openai.com/v1",
"api_key": os.environ["OPENAI_API_KEY"],
"model": "gpt-4o-mini",
},
{
"name": "groq",
"base_url": "https://api.groq.com/openai/v1",
"api_key": os.environ["GROQ_API_KEY"],
"model": "llama-3.1-8b-instant",
},
]Step 2: Implement the failover client
We instantiate one OpenAI client per provider. On RateLimitError or APIError we rotate to the next. Authentication or validation errors bubble up—those are config bugs, not transient failures.
from openai import OpenAI, APIError, RateLimitError
class FailoverChat:
def __init__(self, providers):
self.clients = [
(p, OpenAI(base_url=p["base_url"], api_key=p["api_key"]))
for p in providers
]
def complete(self, messages, **kwargs):
last_err = None
for p, client in self.clients:
try:
resp = client.chat.completions.create(
model=p["model"], messages=messages, **kwargs
)
return {"provider": p["name"], "response": resp}
except (RateLimitError, APIError) as e:
last_err = e
continue
raise RuntimeError(f"All providers failed: {last_err}")Instantiate and run a smoke test:
chat = FailoverChat(PROVIDERS)
out = chat.complete([{"role": "user", "content": "Ping"}])
print(out["provider"], out["response"].choices[0].message.content)Expected output (truncated):
openai Pong! How can I help you today?If OpenAI is returning 429s in your region, you’ll see groq printed without changing a line of calling code.
Streaming caveat
Streaming complicates failover: you cannot switch providers mid-token. Either buffer the full response before yielding to the agent, or restart the stream on the next provider if the first fails before the first token. For most agent loops, non-streaming calls are simpler and the latency gap is acceptable.
Step 3: Forward cache-control and routing hints
Production agents reuse prompts—system instructions, few-shot examples, tool schemas. Provider caching cuts cost and tail latency. The OpenAI-compatible spec passes cache hints via extra_headers. Our **kwargs already forwards them, but be explicit in your agent code:
out = chat.complete(
messages=[
{"role": "system", "content": "You are a terse ops bot."},
{"role": "user", "content": "Status?"},
],
extra_headers={"cache-control": "max-age=300"},
temperature=0,
)If a provider ignores the header, the call still succeeds. If you later switch to a gateway that honors client routing directives, the same header works unchanged.
Step 4: Wrap it in a minimal agent loop with tools
A real agent loops: the model emits a tool call, you execute it, you feed the result back. Failover must survive across iterations, not just the first turn. Below is a stripped-down ReAct loop with a dummy weather tool.
def fake_weather(lat: float, lon: float) -> str:
return "22C, clear"
TOOLS = [{
"type": "function",
"function": {
"name": "fake_weather",
"parameters": {
"type": "object",
"properties": {"lat": {"type": "number"}, "lon": {"type": "number"}},
"required": ["lat", "lon"],
},
},
}]
def run_agent(query: str):
messages = [{"role": "user", "content": query}]
for _ in range(5):
out = chat.complete(messages, tools=TOOLS, tool_choice="auto")
resp = out["response"]
msg = resp.choices[0].message
messages.append(msg)
if not msg.tool_calls:
return {"provider": out["provider"], "content": msg.content}
for call in msg.tool_calls:
if call.function.name == "fake_weather":
args = json.loads(call.function.arguments)
result = fake_weather(**args)
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": result,
})
return {"provider": "none", "content": "loop exhausted"}The key point: chat.complete is the only LLM entrypoint. Every iteration gets failover for free, and the provider field tells you which backend actually answered.
Step 5: Force a provider failure to prove fallback
Don’t wait for a real outage. Inject a broken key into the first provider and watch the chain rotate.
PROVIDERS[0]["api_key"] = "sk-broken"
chat = FailoverChat(PROVIDERS)
out = chat.complete([{"role": "user", "content": "Ping"}])
print(out["provider"]) # -> groqYou should see groq printed. Restore the key afterward. In a load test, script this by mocking client.chat.completions.create to raise RateLimitError for the first N calls.
Step 6: When to delegate failover to a gateway
Maintaining the provider list, health checks, and token accounting inside every service is repetitive. If you’d rather not operate that logic, n4n.ai provides one OpenAI-compatible endpoint that addresses 240+ models and performs automatic fallback when a provider is rate-limited or degraded. You point a single client at the gateway and keep the same extra_headers routing hints.
from openai import OpenAI
gateway = OpenAI(
base_url=os.environ["N4N_BASE_URL"],
api_key=os.environ["N4N_API_KEY"],
)
# No provider list needed; fallback is server-side.
resp = gateway.chat.completions.create(
model="auto", # gateway picks healthy backend
messages=[{"role": "user", "content": "Ping"}],
extra_headers={"cache-control": "max-age=300"},
)Per-token usage metering comes back in the standard usage field, so your existing billing code works unchanged.
Operational checklist
- Set timeouts on the client (
timeout=10) so a hung provider doesn’t block the agent. - Log
providerfrom each response; dashboards should show failover rate. - Alert when the last provider in the chain is hit—it means primary is unhealthy.
- Rotate keys via env, never hardcode.
- For stateful agents, persist
messagesso a failover mid-loop doesn’t lose context. - Add a synthetic canary call in your deploy pipeline that forces a failover to verify the chain.
That’s the whole multi-provider failover tutorial. The pattern is roughly twenty lines of Python, but it’s the difference between a demo and a system that survives a vendor incident at 3 a.m.