A 429 from an LLM provider can stall an agent mid-task. Building rate limit routing agents that detect throttling and shift traffic to healthy endpoints is the difference between a demo and a production system.
Most teams start with a single openai call wrapped in a retry loop. That breaks the moment your primary provider enforces a per-minute token quota or a concurrent request cap. This guide walks through a concrete fallback chain, backoff strategy, and circuit breaker you can ship today.
Why naive retries fail
A bare time.sleep(1) retry ignores the Retry-After header and treats all providers as equal. If the same model is saturated across regions, you burn latency and still fail. Rate limit routing agents need to inspect the failure, parse headers, and switch models or providers without blocking the agent loop.
Step 1: Capture the 429 and its metadata
OpenAI-compatible APIs return a 429 with a Retry-After or x-ratelimit-reset header. Catch the exception and extract these fields before deciding what to do.
from openai import OpenAI, APIStatusError
import os
def call_model(base_url, api_key, model, messages):
client = OpenAI(base_url=base_url, api_key=api_key)
try:
resp = client.chat.completions.create(model=model, messages=messages)
return resp
except APIStatusError as e:
if e.status_code == 429:
retry_after = e.response.headers.get("Retry-After")
reset = e.response.headers.get("x-ratelimit-reset")
raise RateLimitHit(retry_after, reset)
raise
class RateLimitHit(Exception):
def __init__(self, retry_after, reset):
self.retry_after = int(retry_after) if retry_after else None
self.reset = reset
super().__init__("429")Log the model, base URL, and reset time. That data feeds the routing decision in later steps.
Step 2: Build a provider fallback chain
Define an ordered list of targets. Each target is an OpenAI-compatible endpoint and model. A gateway such as n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and applies automatic fallback when a provider is rate-limited or degraded, but you can also manage the chain yourself.
PROVIDERS = [
{"base_url": "https://api.openai.com/v1", "api_key": os.environ["OAI"], "model": "gpt-4o-mini"},
{"base_url": "https://api.anthropic.com/v1", "api_key": os.environ["ANT"], "model": "claude-3-haiku"},
{"base_url": "https://gateway.n4n.ai/v1", "api_key": os.environ["N4N"], "model": "meta-llama/llama-3.1-8b"},
]The router iterates the list, skipping providers that are circuit-open (see Step 4).
def route(messages, providers):
last_err = None
for p in providers:
if breaker[p["base_url"]].is_open():
continue
try:
return call_model(p["base_url"], p["api_key"], p["model"], messages)
except RateLimitHit as e:
breaker[p["base_url"]].trip(e.retry_after)
last_err = e
raise last_err or RuntimeError("no providers")Step 3: Implement exponential backoff with jitter
When a provider returns a Retry-After, honor it. For unspecified 429s, use capped exponential backoff. Never busy-loop.
import random, time
def backoff(attempt, explicit=None):
if explicit:
time.sleep(min(explicit, 30))
return
sleep = min(2 ** attempt + random.uniform(0, 0.5), 30)
time.sleep(sleep)Wrap the route call in a loop with a max attempt budget per agent step:
def agent_step(messages, max_attempts=4):
for attempt in range(max_attempts):
try:
return route(messages, PROVIDERS)
except RateLimitHit as e:
backoff(attempt, e.retry_after)
raise RuntimeError("exhausted routing attempts")Step 4: Add a circuit breaker to skip degraded providers
A provider that 429s repeatedly should be avoided for a short window. An in-memory breaker keyed by base URL works for single-process agents; use Redis for distributed workers.
from collections import defaultdict
class Breaker:
def __init__(self, cooldown=60):
self.cooldown = cooldown
self.open_until = 0
def trip(self, retry_after):
self.open_until = time.time() + (retry_after or self.cooldown)
def is_open(self):
return time.time() < self.open_until
breaker = defaultdict(Breaker)This prevents your rate limit routing agents from hammering a dead endpoint while other providers absorb load.
Step 5: Honor routing directives and cache hints
Production agents often need to pin a model for consistency or leverage provider prompt caching. If you send cache_control hints, ensure your client forwards them. Some gateways honor client routing directives and forward provider cache-control hints, so a single request can specify model: claude-3-sonnet and still get cached prefix handling without custom middleware.
When self-managing, pass extra headers explicitly:
client = OpenAI(base_url=base_url, api_key=api_key, default_headers={
"x-cache-control": "ttl=300"
})Verify your target provider actually supports the hint; otherwise it is silently ignored.
Step 6: Meter usage and observe
Per-token usage metering is non-negotiable for cost control. Capture usage from each response and tag it with the provider that served it.
def log_usage(resp, provider):
u = resp.usage
metrics.record({
"provider": provider["base_url"],
"model": resp.model,
"prompt_tokens": u.prompt_tokens,
"completion_tokens": u.completion_tokens,
})If you use a gateway with per-token usage metering, aggregate from its response headers or dashboard instead of instrumenting each branch.
Verify your agent routes around limits
You cannot trust fallback logic until you force a 429. Stand up a local mock that returns 429 for the first three calls, then 200.
# test_mock.py
from fastapi import FastAPI, Response
import pytest
app = FastAPI()
calls = {"n": 0}
@app.post("/v1/chat/completions")
async def mock(resp: Response):
calls["n"] += 1
if calls["n"] <= 3:
resp.status_code = 429
resp.headers["Retry-After"] = "0"
return {"error": "rate limited"}
return {"model": "mock", "choices": [{"message": {"content": "ok"}}],
"usage": {"prompt_tokens": 1, "completion_tokens": 1}}Point PROVIDERS[0] at this mock and assert agent_step returns a valid response after the limit clears. Run:
pytest test_mock.py -qIf the test passes, your rate limit routing agents correctly detect the 429, back off, and fall through to the next provider. In production, watch the breaker trip counts and token metrics; a healthy system shows occasional trips but zero agent-step failures.
Closing notes
Routing around 429s is not glamorous, but it is the core reliability work for agentic apps. Capture the headers, chain providers, break circuits, and meter everything. Do that and your agents will survive the next provider outage without a page at 3 a.m.