When you build applications that call multiple LLM APIs, you will eventually hit rate limits. A pragmatic way to handle 429 rate limit llm fallback is to treat a 429 as a signal to shift traffic to a secondary provider instead of blocking the user. This post walks through a concrete client implementation that retries with backoff and falls back across providers, then shows how to verify it under test.
Step 1: Model providers behind a single interface
The first mistake engineers make is writing fallback logic inline for each SDK. That scatters error mapping and makes the chain brittle. Define one abstract interface and adapt each vendor client to it.
from abc import ABC, abstractmethod
class ChatCompletionProvider(ABC):
@abstractmethod
def complete(self, messages: list[dict], model: str) -> str:
"""Return assistant text or raise RateLimitError / ProviderError."""Now wrap OpenAI. The official openai package raises openai.RateLimitError on HTTP 429. Catch it and re-raise a domain error:
import openai
from openai import RateLimitError as OpenAIRateLimitError
class OpenAIProvider(ChatCompletionProvider):
def __init__(self, api_key: str):
self.client = openai.OpenAI(api_key=api_key)
def complete(self, messages, model):
try:
resp = self.client.chat.completions.create(model=model, messages=messages)
return resp.choices[0].message.content
except OpenAIRateLimitError as e:
raise RateLimitError(_parse_retry_after(e)) from eAnthropic’s SDK uses a different exception type but the same HTTP semantics. The point is that your application code never sees vendor-specific errors.
Mapping errors correctly
Do not fall back on every exception. A 400 from malformed input should not trigger a provider switch. Only 429 (and optionally 503) maps to RateLimitError. Everything else bubbles up.
Step 2: Classify 429s and extract Retry-After
A 429 is not a monolith. Some providers send a Retry-After header (in seconds or HTTP date). If it is small, you can retry the same provider; if it is large, skip to the next one.
def _parse_retry_after(exc) -> float | None:
response = getattr(exc, "response", None)
if not response:
return None
headers = getattr(response, "headers", {})
ra = headers.get("retry-after")
if ra and ra.isdigit():
return float(ra)
return None
class RateLimitError(Exception):
def __init__(self, retry_after: float | None = None):
self.retry_after = retry_after
super().__init__("rate limited")This helper keeps the fallback decision data-driven. If retry_after is None, assume the limit is indefinite and move on.
Step 3: Implement the ordered fallback chain
You should keep an explicit ordered list of providers reflecting your cost and latency preferences. The loop below is the core of any script that must handle 429 rate limit llm fallback without user-visible failures.
import time
def handle_429_rate_limit_llm_fallback(providers, messages, model):
last_exc = None
for provider in providers:
try:
return provider.complete(messages, model)
except RateLimitError as e:
if e.retry_after and e.retry_after <= 2.0:
time.sleep(e.retry_after)
try:
return provider.complete(messages, model)
except RateLimitError as e2:
last_exc = e2
continue
last_exc = e
continue
raise last_exc or RuntimeError("all providers failed")Notice we only retry the same provider when retry_after is tiny. Otherwise we burn latency waiting for a provider that is clearly saturated. Non-rate-limit exceptions are intentionally not caught here.
Step 4: Add jittered exponential backoff for transient limits
Even within a single provider, a brief burst can trigger a 429 that clears in milliseconds. A small backoff wrapper reduces fallback churn.
import random
def backoff_retry(fn, max_attempts=3, base=0.5):
for attempt in range(max_attempts):
try:
return fn()
except RateLimitError as e:
if attempt == max_attempts - 1:
raise
wait = (base * (2 ** attempt)) + random.uniform(0, 0.2)
time.sleep(wait)Wire it into the provider adapter or the chain. The jitter prevents thundering-herd retries when many workers hit the same limit simultaneously.
Step 5: Protect the chain with a circuit breaker
If a provider is consistently returning 429, calling it on every request wastes round-trip time. A simple in-memory breaker tracks consecutive failures:
class CircuitBreaker:
def __init__(self, threshold=5, cooldown=30):
self.failures = 0
self.threshold = threshold
self.cooldown = cooldown
self.opened_at = 0.0
def allow(self) -> bool:
if self.failures >= self.threshold:
if time.time() - self.opened_at > self.cooldown:
self.failures = 0
return True
return False
return True
def record_failure(self):
self.failures += 1
self.opened_at = time.time()
def record_success(self):
self.failures = 0Check breaker.allow() before invoking a provider, and record the outcome after. In a multi-process deployment, use Redis instead of process memory; the pattern stays identical.
Step 6: Offload fallback to a gateway (optional)
Hand-rolling the above is instructive, but maintaining provider adapters and health state is ongoing toil. A gateway like n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and performs automatic fallback when a provider is rate-limited or degraded. You send a standard request and it honors client routing directives and forwards provider cache-control hints:
import openai
client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")
client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)Behind that call, the gateway implements the same fallback and backoff logic described above. Use it when your team would rather spend cycles on product features than on rate-limit plumbing.
Step 7: Verify your fallback logic
A fallback chain is only as good as its tests. Start with a unit test that simulates a provider failing with a 429 and a second provider succeeding.
class FailingProvider(ChatCompletionProvider):
def __init__(self, fail_times=1):
self.fail_times = fail_times
def complete(self, messages, model):
if self.fail_times > 0:
self.fail_times -= 1
raise RateLimitError(0.0)
return "ok"
class OkProvider(ChatCompletionProvider):
def complete(self, messages, model):
return "fallback-worked"
def test_fallback():
providers = [FailingProvider(fail_times=1), OkProvider()]
result = handle_429_rate_limit_llm_fallback(providers, [], "test")
assert result == "fallback-worked"For integration confidence, run a local mock server that returns 429 for the first N requests, then 200. Point your first provider at it and watch the chain shift. A quick bash check with curl against such a mock:
curl -i -X POST http://localhost:8080/v1/chat -d '{"model":"x"}' -H 'content-type: application/json'
# expect: HTTP/1.1 429 followed by 200 on retryHow to verify success in production
Instrument two counters: llm_requests_total{provider} and llm_fallback_total. Success means a request that would have errored returns a valid completion, and llm_fallback_total increments exactly when the primary provider emits a 429. If you never see fallbacks during load tests, your rate limits are likely set too high to validate the path—temporarily lower the limit or inject faults with a proxy like Toxiproxy.
The pattern to handle 429 rate limit llm fallback is not exotic: unify the interface, classify the error, chain with backoff, guard with a breaker, and prove it with fault injection. Do that and your users stop seeing “too many requests” no matter which model backend hiccups.