When a single LLM provider throws a 429 or a 503, your production traffic shouldn’t wait. A retry then fallback chain llm api pattern lets you absorb transient provider errors by retrying with backoff and then shifting the request to a secondary provider—without blocking the caller. This post walks through building that chain in Python with the OpenAI SDK, so you can ship resilient inference calls today.
Step 1: Classify errors and order your providers
Not every error deserves a retry. Rate limits (HTTP 429) and upstream 5xx responses are transient. Authentication failures (401), invalid requests (400), and context-length violations are permanent for that exact payload. Retry only the transient set; fall back when retries exhaust or a permanent error hits on the primary.
Define your provider list explicitly. Order matters: put the cheapest or lowest-latency provider first, reserve premium models for fallback.
PROVIDERS = [
{"base_url": "https://api.openai.com/v1", "api_key": "sk-...", "model": "gpt-4o-mini"},
{"base_url": "https://api.anthropic.com/v1", "api_key": "sk-ant-...", "model": "claude-3-5-sonnet"},
{"base_url": "https://local-vllm:8000/v1", "api_key": "EMPTY", "model": "mistral-7b"},
]Each entry is an OpenAI-compatible endpoint. The same client code works across vendors because they expose the /chat/completions shape.
Step 2: Implement retry with exponential backoff
Retry logic must be bounded and jittery. Fixed intervals synchronize failures across concurrent callers; exponential backoff with random jitter spreads them out.
from openai import OpenAI, RateLimitError, APIError
import time, random
def complete_with_retry(client, model, messages, max_retries=3):
attempt = 0
while attempt < max_retries:
try:
return client.chat.completions.create(model=model, messages=messages)
except RateLimitError:
pass # 429, always retryable
except APIError as e:
if not (e.status_code and 500 <= e.status_code < 600):
raise # permanent, don't retry
wait = (2 ** attempt) + random.uniform(0, 1)
time.sleep(wait)
attempt += 1
raise RuntimeError("retry budget exhausted")The function returns the successful response object or raises after max_retries. It never catches non-transient errors, so a malformed request fails fast instead of burning retries.
Step 3: Build the fallback loop
Wrap the retry function in a loop over providers. Capture the last exception so you can surface a meaningful error if every provider fails.
def retry_then_fallback_chain_llm_api(messages, providers=PROVIDERS):
last_err = None
for cfg in providers:
client = OpenAI(base_url=cfg["base_url"], api_key=cfg["api_key"])
try:
resp = complete_with_retry(client, cfg["model"], messages)
return resp
except Exception as e:
last_err = e
continue
raise last_errThis is the core of the retry then fallback chain llm api pattern. The caller gets a single response object or a single exception; it doesn’t need to know which provider served the token.
Step 4: Pass cache and routing hints
Production gateways often accept headers to control caching and provider selection. If you front your providers with a unified endpoint, forward cache-control hints so repeated prompts hit provider caches.
def complete_with_retry(client, model, messages, max_retries=3):
attempt = 0
while attempt < max_retries:
try:
return client.chat.completions.create(
model=model,
messages=messages,
extra_headers={
"x-cache-control": "max-age=3600",
"x-provider-preference": "openai,anthropic"
}
)
except RateLimitError:
pass
except APIError as e:
if not (e.status_code and 500 <= e.status_code < 600):
raise
time.sleep((2 ** attempt) + random.uniform(0, 1))
attempt += 1
raise RuntimeError("retry budget exhausted")Some gateways, including n4n.ai, honor client routing directives and forward provider cache-control hints, so you can pin a model or force a specific provider via headers while still getting fallback across 240+ models behind one OpenAI-compatible endpoint.
Step 5: Meter usage and log attempts
Per-token cost visibility is non-negotiable when you span multiple providers. Extract usage from the response and tag it with the provider that succeeded.
def retry_then_fallback_chain_llm_api(messages, providers=PROVIDERS):
last_err = None
for cfg in providers:
client = OpenAI(base_url=cfg["base_url"], api_key=cfg["api_key"])
try:
resp = complete_with_retry(client, cfg["model"], messages)
usage = resp.usage.model_dump()
print({"provider": cfg["base_url"], "model": cfg["model"], **usage})
return resp
except Exception as e:
last_err = e
continue
raise last_errIf you use a gateway that provides per-token usage metering, the same usage object reflects the underlying provider’s counts without extra plumbing.
Step 6: Verify the chain end to end
A chain is only as good as its test. Mock the client to fail the first provider twice with RateLimitError, then succeed. Assert the second provider is never touched.
import pytest
from openai import RateLimitError
from your_module import complete_with_retry, retry_then_fallback_chain_llm_api
class FakeCompletions:
def __init__(self, fails):
self.fails = fails
self.calls = 0
def create(self, **kwargs):
self.calls += 1
if self.calls <= self.fails:
raise RateLimitError("rate", response=None, body=None)
class R:
usage = type("U", (), {"prompt_tokens":1,"completion_tokens":1,"total_tokens":2})()
return R()
class FakeClient:
def __init__(self, fails):
self.chat = type("Chat", (), {"completions": FakeCompletions(fails)})()
def test_retry_then_fallback(monkeypatch):
primary = FakeClient(fails=2)
secondary = FakeClient(fails=99)
providers = [
{"base_url": "p1", "api_key": "x", "model": "m1"},
{"base_url": "p2", "api_key": "x", "model": "m2"},
]
monkeypatch.setattr("your_module.OpenAI", lambda **kw: primary if kw["base_url"]=="p1" else secondary)
resp = retry_then_fallback_chain_llm_api([{"role":"user","content":"hi"}], providers)
assert primary.chat.completions.calls == 3
assert secondary.chat.completions.calls == 0Run pytest -q and confirm the primary is retried three times (initial plus two retries) and the secondary is never called. Then run an integration test with a deliberately invalid API key on the first provider to force fallback to the second, and watch the usage log print the fallback provider.
Operational notes
Set a hard timeout on the client (timeout=30) so a hung connection doesn’t stall the chain. Use a circuit breaker if a provider shows repeated degradation—skip it for a cooldown period instead of always hitting it first. Keep max_retries low (2–3); fallback is cheaper than long retry storms.
The retry then fallback chain llm api design keeps your service alive when individual providers flap. Implement the steps above, test with mocks, and you have production-grade resilience without a heavyweight framework.