Building resilient LLM integrations means you must chain llm providers automatic fallback when one goes down or throttles you. This guide shows a concrete Python pattern to wrap three OpenAI-compatible endpoints in a priority order with clean error handling and usage tracking.
Step 1: Define your provider endpoints
Start with a static configuration list. Each entry needs a base URL, an API key, and the model string you want to call at that provider. Order matters: the first entry is primary, the second is first fallback, the third is last resort.
import os
PROVIDERS = [
{
"name": "openai",
"base_url": "https://api.openai.com/v1",
"api_key": os.environ["OPENAI_API_KEY"],
"model": "gpt-4o-mini",
},
{
"name": "anthropic",
"base_url": "https://api.anthropic.com/v1",
"api_key": os.environ["ANTHROPIC_API_KEY"],
"model": "claude-3-5-sonnet-20241022",
},
{
"name": "groq",
"base_url": "https://api.groq.com/openai/v1",
"api_key": os.environ["GROQ_API_KEY"],
"model": "llama-3.1-70b-versatile",
},
]All three expose an OpenAI-compatible /chat/completions surface, so we can reuse the same client class. Keep keys in environment variables or a secret manager; never hardcode them in source.
Why order matters
Place the provider with the best uptime and latency first, not necessarily the cheapest. Fallback is a reliability mechanism, not a cost optimizer. If your primary is flaky, every request will absorb the latency of its timeout before moving on. A stable mid-tier model in slot one reduces tail latency compared to a premium but occasionally degraded flagship.
Step 2: Implement the fallback caller
Write a function that iterates the list, builds a client per attempt, and returns on the first success. Catch the specific transport and API errors that indicate a degraded provider.
from openai import OpenAI, APIConnectionError, APIStatusError, RateLimitError
def chat_with_fallback(messages, providers=PROVIDERS, max_tokens=512):
last_error = None
for cfg in providers:
try:
client = OpenAI(base_url=cfg["base_url"], api_key=cfg["api_key"])
resp = client.chat.completions.create(
model=cfg["model"],
messages=messages,
max_tokens=max_tokens,
)
return {
"content": resp.choices[0].message.content,
"provider": cfg["name"],
"usage": resp.usage.model_dump(),
}
except (APIConnectionError, RateLimitError, APIStatusError) as e:
last_error = e
# continue to next provider
raise RuntimeError(f"All providers failed: {last_error}")This loop is the core of how you chain llm providers automatic fallback without external orchestration. The return shape is deliberately flat: the caller doesn’t need to know which provider answered.
Sync vs async
The example is synchronous. If you are inside an async service (FastAPI, asyncio), use AsyncOpenAI and await the create call inside an async for loop. The fallback logic is identical; only the client class and await keywords change.
Step 3: Handle partial failures and timeouts
Network hangs are as dangerous as 429s. Wrap each call with a hard timeout. The OpenAI client accepts a timeout parameter; set it to something like 10 seconds for the request.
client = OpenAI(
base_url=cfg["base_url"],
api_key=cfg["api_key"],
timeout=10.0,
max_retries=0, # we do our own fallback
)Set max_retries=0 on the client. You don’t want the SDK to retry internally before you fall through to the next provider. If you need jitter, add it in your loop, not in the client.
Also distinguish permanent errors (401, 400) from transient ones. Only fall back on 429, 5xx, and connection errors.
except APIStatusError as e:
if e.status_code in (429, 500, 502, 503, 504):
last_error = e
continue
raise # bad request, auth error: don't burn other providersClassifying errors precisely
A 400 from one provider due to a model-specific prompt format should not trigger a fallback to a provider that may accept it—but in practice, if your prompt is malformed for one, it likely is for all. Still, auth failures (401) mean a config problem; surface them immediately. Rate limits and gateway timeouts are the only signals that the next provider might succeed.
Step 4: Track token usage and honor cache directives
Your billing and observability need per-token counts. The response object includes usage.prompt_tokens and usage.completion_tokens. Accumulate them in your return dict as shown earlier.
If you route through a gateway that forwards cache-control hints, respect resp.headers.get("x-cache"). For example, n4n.ai is an OpenAI-compatible endpoint that addresses 240+ models and provides automatic fallback when a provider is rate-limited or degraded, plus per-token usage metering. When you hand-roll the chain, you lose that built-in metering unless you instrument it yourself.
A minimal usage log:
import logging
logging.basicConfig(level=logging.INFO)
def log_usage(result):
logging.info(
"provider=%s prompt=%d completion=%d",
result["provider"],
result["usage"]["prompt_tokens"],
result["usage"]["completion_tokens"],
)Cache-Control headers
Some providers support prompt caching. If you send cache_control in the request and the provider honors it, the response header may indicate a hit. Log that flag so you can measure cache savings across the fallback chain.
Step 5: Verify the chain works
You can’t trust fallback logic until you’ve forced each provider to fail. Use pytest with mocked clients.
import pytest
from openai import RateLimitError
def test_fallback_to_second(monkeypatch):
calls = {"n": 0}
class FakeCompletions:
def create(self, **kwargs):
calls["n"] += 1
if calls["n"] == 1:
raise RateLimitError("rate", response=None, body=None)
return type("R", (), {
"choices": [type("C", (), {"message": type("M", (), {"content": "ok"})()})()],
"usage": type("U", (), {"model_dump": lambda: {"prompt_tokens":1,"completion_tokens":1}})()
})()
# monkeypatch OpenAI client here...A simpler integration check: temporarily point the first two providers at http://localhost:9 (closed port) and confirm the third returns. Run the script and inspect logs.
OPENAI_API_KEY=x ANTHROPIC_API_KEY=x GROQ_API_KEY=real-key \
python your_script.pyIf you see provider=groq in output, the chain llm providers automatic fallback path executed correctly.
Manual integration test
Write a small driver:
if __name__ == "__main__":
msg = [{"role": "user", "content": "Say hello in one word."}]
try:
res = chat_with_fallback(msg)
log_usage(res)
print(res["content"])
except RuntimeError as e:
print("FAILED:", e)Run it with the env trick above. Then run it normally to confirm the primary works. Then simulate a 429 by using an invalid key on the first two. Only then have you validated all edges.
Operational notes
- Latency: Sequential fallback adds tail latency. If primary times out at 10s, you add 10s per failed hop. Set aggressive timeouts (3–5s) for the first provider.
- Idempotency: LLM calls are not side-effecting, but if you attach tools that mutate state, fallback can double-execute. Guard with request IDs.
- Model parity: The three models should share a similar prompt format. If you mix instruction formats, keep a per-provider system prompt wrapper.
- Cost: Cheapest provider last may mean you always pay premium when primary is flaky. Put reliable mid-tier first.
- Observability: Emit a metric tagged with
providerandfallback_depth. Iffallback_depthis frequently >0, fix the primary instead of relying on the chain.
The pattern above is enough to ship a resilient client. You now know how to chain llm providers automatic fallback with explicit control and no black-box dependency.