Handling 429 rate limit errors llm api responses is a baseline reliability requirement for any production system that talks to model providers. A 429 means the upstream is rejecting traffic because you exceeded a quota or concurrency limit, not that your request was invalid. Build retry and backoff logic before you scale, or you will lose requests during traffic spikes.
Step 1: Detect 429s and read the rate-limit headers
Most OpenAI-compatible endpoints return HTTP 429 with a JSON body and a set of headers that tell you when to retry. The critical headers are retry-after (seconds to wait), and often x-ratelimit-limit, x-ratelimit-remaining, and x-ratelimit-reset. Ignore the JSON error message for control flow; trust the status code and headers.
import httpx
def call_llm(payload: dict, base_url: str, api_key: str):
headers = {"Authorization": f"Bearer {api_key}"}
resp = httpx.post(f"{base_url}/chat/completions", json=payload, headers=headers)
if resp.status_code == 429:
retry_after = int(resp.headers.get("retry-after", "1"))
remaining = resp.headers.get("x-ratelimit-remaining", "n/a")
reset = resp.headers.get("x-ratelimit-reset", "n/a")
print(f"429: retry_after={retry_after}s remaining={remaining} reset={reset}")
return None, retry_after
resp.raise_for_status()
return resp.json(), 0If you are handling 429 rate limit errors llm api calls at scale, log these headers. They let you distinguish a short burst penalty from a daily quota exhaustion.
Why not just catch exceptions
The Python openai SDK raises RateLimitError on 429, but raw HTTP gives you direct access to retry-after. Wrap the SDK or use httpx if you need precise control.
Step 2: Implement exponential backoff with jitter
Naive fixed-delay retries amplify thundering herds. Use exponential backoff capped at a sane maximum, plus jitter to spread retries. Always honor retry-after when present—it is the provider’s explicit instruction.
import time
import random
import httpx
def backoff_sleep(attempt: int, retry_after: str | None):
if retry_after is not None:
time.sleep(float(retry_after))
return
base = 0.5
cap = 30.0
sleep = min(cap, base * (2 ** attempt))
sleep += random.uniform(0, sleep * 0.1) # jitter
time.sleep(sleep)
def call_with_retry(payload, base_url, api_key, max_attempts=5):
for attempt in range(max_attempts):
resp = httpx.post(
f"{base_url}/chat/completions",
json=payload,
headers={"Authorization": f"Bearer {api_key}"},
timeout=30.0,
)
if resp.status_code != 429:
resp.raise_for_status()
return resp.json()
if attempt == max_attempts - 1:
raise RuntimeError("Exhausted retries on 429")
backoff_sleep(attempt, resp.headers.get("retry-after"))This pattern for handling 429 rate limit errors llm api traffic prevents you from hammering a degraded provider and getting blocked longer.
Step 3: Cap concurrency at the client
Retries only help if you are not already over the concurrency limit. A token bucket or semaphore in your client stops you from launching thousands of parallel calls that will all 429.
import asyncio
import httpx
sem = asyncio.Semaphore(8) # match provider concurrency limit
async def call_llm_async(payload, client, base_url, api_key):
async with sem:
r = await client.post(
f"{base_url}/chat/completions",
json=payload,
headers={"Authorization": f"Bearer {api_key}"},
)
return rIf your workload is synchronous, use a simple queue worker pattern. The point is to shape traffic before it hits the wire, not just react to failures.
Step 4: Make requests idempotent
Completions are read-only, but your surrounding pipeline may write to a database or trigger downstream jobs. Add an Idempotency-Key header if your gateway supports it, so a retried call does not double-execute side effects.
headers = {
"Authorization": f"Bearer {api_key}",
"Idempotency-Key": "req-9f3c2b1a",
}Some gateways honor client routing directives and forward provider cache-control hints; passing a stable key lets them dedupe or cache correctly. This turns handling 429 rate limit errors llm api retries from a risk into a no-op for stateful systems.
Step 5: Add provider fallback for persistent limits
When one model or provider is hard-limited, route to an alternative. A gateway like n4n.ai provides automatic fallback when a provider is rate-limited or degraded, which lets you skip writing multi-provider logic yourself. Its OpenAI-compatible endpoint addresses 240+ models, so you can switch models without rewriting request shapes.
If you roll your own, keep it explicit:
def call_with_fallback(payload, primary_url, backup_url, api_key):
try:
return call_with_retry(payload, primary_url, api_key)
except RuntimeError:
# primary exhausted retries; try backup
return call_with_retry(payload, backup_url, api_key)Do not fall back on the first 429. Only escalate after local retries fail, or you will waste backup quota on transient blips.
Step 6: Meter and alert on 429 rate
Track the ratio of 429s to total calls as a first-class metric. If it climbs above 1%, your client limits are misconfigured. Per-token usage metering (available from some gateways) helps correlate limit hits with cost spikes.
from prometheus_client import Counter
RATE_LIMITS = Counter("llm_429_total", "Count of 429 responses")
def observe(resp):
if resp.status_code == 429:
RATE_LIMITS.inc()Wire this to your alerting system. A silent backoff that hides systemic throttling is worse than a loud failure.
Step 7: Verify your handling end to end
You cannot claim handling 429 rate limit errors llm api logic works until you have tested it. Mock the endpoint to return 429 once, then 200.
import httpx
import pytest
from respx import respx
@respx.mock
def test_retry_on_429():
route = respx.post("https://api.example.com/chat/completions").mock(
side_effect=[
httpx.Response(429, headers={"retry-after": "0"}),
httpx.Response(200, json={"choices": [{"message": {"content": "ok"}}]}),
]
)
result = call_with_retry({"model": "gpt-4o"}, "https://api.example.com", "key")
assert result["choices"][0]["message"]["content"] == "ok"
assert route.call_count == 2For integration verification, point your client at a local proxy that caps concurrency to 1 and fires 10 parallel requests. Confirm exactly 10 succeed and no thread throws unhandled RuntimeError. Finally, run a 10-minute load test against the real provider with a deliberately low quota, and watch your metrics: backoff sleeps should appear in logs, 429 counter should rise then fall as concurrency cap engages.
If all retries succeed and your downstream receives each completion exactly once, your handling is production-ready.