When you build against LLM providers, you will eventually need to handle 429 rate limit error llm api responses. The naive retry-immediately pattern burns quotas and amplifies outages; a disciplined approach treats 429 as a backpressure signal, not a failure. This guide walks through concrete steps to make your client resilient without adding fragility.
Step 1: Detect the 429 correctly
A 429 is the HTTP status code for “Too Many Requests”. Most LLM REST APIs return it from the edge, but the error payload shape varies by vendor. OpenAI’s Python SDK raises openai.APIStatusError with status_code == 429 and a JSON body where error.type is often rate_limit_error. Anthropic’s API returns a similar structure with type: "rate_limit_error". OpenAI-compatible endpoints follow the same convention.
If you use the official SDK:
import openai
from openai import APIStatusError
try:
resp = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "hi"}]
)
except APIStatusError as e:
if e.status_code == 429:
body = e.response.json()
print("Rate limited:", body.get("error", {}).get("type"))With raw requests:
import requests
r = requests.post(
"https://api.openai.com/v1/chat/completions",
headers={"Authorization": "Bearer " + KEY},
json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]}
)
if r.status_code == 429:
err = r.json().get("error", {})
print("429:", err.get("type"), err.get("message"))To handle 429 rate limit error llm api responses uniformly across SDKs, wrap these in a custom RateLimitError exception in your client layer. Do not confuse 429 with 500 or 503; the latter imply server faults and deserve different backoff caps.
Streaming calls complicate detection. Some providers only emit the 429 after the HTTP headers flush, mid-stream. Catch exceptions on the iterator and check status if available.
Step 2: Implement exponential backoff with jitter
Immediate retries hammer the provider and extend the limit window. Exponential backoff waits base * 2**attempt seconds, capped, with random jitter to avoid synchronized retries across distributed workers.
import time
import random
def backoff_sleep(attempt, base=0.5, cap=30.0):
delay = min(cap, base * (2 ** attempt))
jitter = random.uniform(0, delay)
time.sleep(jitter)Wrap the call:
def call_with_retry(fn, max_attempts=5):
for attempt in range(max_attempts):
try:
return fn()
except RateLimitError:
if attempt == max_attempts - 1:
raise
backoff_sleep(attempt)Token-based limits are common: a 429 may mean you hit a per-minute token quota, not request count. Backoff alone won’t help if you keep sending large prompts. Reduce max_tokens or batch less aggressively before retrying. The key to handle 429 rate limit error llm api cleanly is to slow down and shrink payloads, not just wait.
Step 3: Respect provider-specific headers and hints
Vendors send rate-limit metadata in response headers. OpenAI includes x-ratelimit-remaining-requests, x-ratelimit-reset-requests (seconds to reset), and similar for tokens. If present, sleep until reset instead of guessing.
if r.status_code == 429:
reset = r.headers.get("x-ratelimit-reset-requests")
if reset:
time.sleep(float(reset) + 0.1)
else:
backoff_sleep(attempt)Not every provider sends Retry-After. Claude and some open-weight endpoints omit it. Fall back to backoff when headers are missing. Also honor cache-control: if you send cache-control: ephemeral and the endpoint forwards provider cache hints, repeated identical prompts hit cache and avoid rate limits entirely.
When you handle 429 rate limit error llm api from multiple vendors, build a header-normalization map so your retry logic reads reset from any source.
Step 4: Use a gateway with automatic fallback
If your traffic spans multiple model vendors, a single vendor’s 429 often means that vendor is degraded, not that you exceeded a global quota. A gateway like n4n.ai can automate fallback across providers when a 429 indicates provider degradation, since it exposes one OpenAI-compatible endpoint covering 240+ models and triggers automatic fallback when a provider is rate-limited or degraded. Your application code does not change:
client = openai.OpenAI(
base_url="https://api.n4n.ai/v1",
api_key=GATEWAY_KEY
)
# same create() call; fallback happens upstreamYou still need backoff for the gateway’s own aggregate limits, but you stop writing per-vendor retry branches. The gateway also provides per-token usage metering so you can see which model burns quota.
Step 5: Add a circuit breaker for persistent limits
Exponential backoff handles transient spikes. If a model returns 429 for every call over minutes, stop sending and shed load. A circuit breaker flips open after N consecutive failures, then half-opens to test recovery.
class CircuitBreaker:
def __init__(self, threshold=5, cooldown=60):
self.failures = 0
self.threshold = threshold
self.cooldown = cooldown
self.opened_at = 0
def allow(self):
if self.failures >= self.threshold:
if time.time() - self.opened_at > self.cooldown:
self.failures = self.threshold // 2 # half-open
return True
return False
return True
def record_failure(self):
self.failures += 1
if self.failures == self.threshold:
self.opened_at = time.time()
def record_success(self):
self.failures = 0Wire it:
breaker = CircuitBreaker()
def call_with_breaker(fn):
if not breaker.allow():
raise RuntimeError("circuit open")
try:
res = call_with_retry(fn)
breaker.record_success()
return res
except RateLimitError:
breaker.record_failure()
raiseTo handle 429 rate limit error llm api at scale, use per-model isolation: one saturated model should not trip the breaker for all traffic. Combine with bulkheads so each tenant gets its own limit track.
Step 6: Verify your handling
You cannot claim resilience without testing. Stand up a local mock that returns 429 with the headers you expect.
mkdir -p mock_429/v1
echo '{"error":{"type":"rate_limit_error","message":"slow down"}}' > mock_429/v1/chat_completions.json
python -m http.server 8000 --directory mock_429Point your client base URL at http://localhost:8000 and run the integration test. Assert:
- First 429 triggers a sleep > 0.
- After
max_attempts, the error propagates. - Circuit breaker opens after threshold.
- Logs show no tight request loop (check timestamps).
If using a gateway, verify fallback by blocking one provider in test config and confirming a 200 from a secondary. Success means your production code survives a provider outage without code changes.
Step 7: Monitor and alert
Backoff hides problems. Emit metrics: rate_limit_errors_total, retry_count, breaker_opened. Alert when 429 rate exceeds 1% of requests for a model. This catches silent degradation before users notice.
A robust system to handle 429 rate limit error llm api needs detection, backoff, header awareness, fallback, breakers, and verification. Ship the wrapper once, and your application code stays clean.