How to handle rate limit errors in production LLM apps

Rate limits are the most common failure mode when you scale LLM integrations beyond a demo. To handle rate limit errors production LLM apps need explicit retry, fallback, and queueing logic—not just a try/except around the API call.

Step 1: Classify errors and read rate-limit headers

A 429 is not a bug; it is a signal that you exceeded a quota. Providers differ in how they encode details. OpenAI returns a RateLimitError with a response.headers dict; Anthropic returns HTTP 429 with retry-after. Always parse Retry-After (seconds) and any provider-specific headers like x-ratelimit-remaining-tokens or x-ratelimit-reset-requests.

import openai

try:
    client.chat.completions.create(model="gpt-4o", messages=[...])
except openai.RateLimitError as e:
    retry_after = e.response.headers.get("retry-after")
    limit = e.response.headers.get("x-ratelimit-limit-requests")
    remaining = e.response.headers.get("x-ratelimit-remaining-requests")
    print(f"429: retry in {retry_after}s, limit {limit}, left {remaining}")

If the header is missing, assume a short fixed delay. Never treat a 429 as a fatal error in a production loop. Distinguish it from 400 (bad request) and 500 (server error) so your retry policy does not mask real bugs.

Step 2: Implement exponential backoff with jitter

Naive time.sleep(2) is wrong: it synchronizes retries across workers and amplifies load. Use exponential backoff capped at a max, with full jitter. The cap prevents multi-minute waits that blow up latency budgets.

import random, time

def backoff_delay(attempt, base=0.5, cap=30):
    delay = min(cap, base * (2 ** attempt))
    return delay * random.uniform(0.5, 1.5)

for attempt in range(5):
    try:
        return call_llm()
    except RateLimitError as e:
        wait = float(e.response.headers.get("retry-after", backoff_delay(attempt)))
        time.sleep(wait)

For most production services, use the tenacity library with @retry(wait=wait_exponential_jitter(), stop=stop_after_attempt(5)). That keeps the logic declarative and avoids hand-rolled loops. Set stop_after_attempt based on your user-facing timeout, not arbitrary patience.

Step 3: Throttle outbound traffic with a token bucket

Backoff reacts; throttling prevents. Deploy a local token bucket keyed by provider and model. This smooths bursts and respects published RPM/TPM limits. Set the bucket rate 20% below the provider’s stated limit to absorb clock skew and shared tenancy.

import time, threading

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate = rate
        self.capacity = capacity
        self.tokens = capacity
        self.lock = threading.Lock()
        self.last = time.time()

    def consume(self, n=1):
        with self.lock:
            now = time.time()
            self.tokens += (now - self.last) * self.rate
            self.tokens = min(self.tokens, self.capacity)
            self.last = now
            if self.tokens >= n:
                self.tokens -= n
                return True
            return False

Call consume() before each request. If false, sleep briefly. In async apps, use an equivalent non-blocking check:

async def consume_async(bucket, n=1):
    while not bucket.consume(n):
        await asyncio.sleep(0.05)

Without throttling, a traffic spike will trigger 429s that backoff only delays, not avoids. The bucket is your first line of defense.

Step 4: Configure fallback models and providers

When a model is consistently limited, switch to a cheaper or less congested one. Define a priority list and catch the rate limit to rotate.

MODELS = ["gpt-4o", "gpt-4o-mini", "claude-3-haiku"]

for model in MODELS:
    try:
        return client.chat.completions.create(model=model, messages=msg)
    except RateLimitError:
        continue
raise RuntimeError("all models rate-limited")

A gateway like n4n.ai exposes one OpenAI-compatible endpoint for 240+ models and applies automatic fallback when a provider is rate-limited or degraded, which removes one class of errors—but client-side throttling remains necessary. If your gateway honors client routing directives, pin a specific provider or region to avoid shared public quotas that you do not control.

Step 5: Forward cache-control hints to cut token volume

Prompt caching reduces repeated token spend and therefore rate-limit pressure. If your client library supports cache_control on system blocks, set it. Gateways that forward provider cache-control hints pass this through transparently.

client.chat.completions.create(
    model="claude-3-5-sonnet",
    messages=[
        {"role": "system", "content": "Long static instructions...",
         "extra_body": {"cache_control": {"type": "ephemeral"}}}
    ]
)

Less token throughput per request means you hit TPM limits later. This is a structural fix, not a band-aid. Combine caching with short prompts: every cached token is a token you do not meter against your limit.

Step 6: Instrument every 429

You cannot operate what you cannot see. Emit a counter per model/provider and a histogram of retry counts. In Prometheus:

from prometheus_client import Counter, Histogram

RATE_LIMITS = Counter("llm_rate_limits_total", "429s", ["model", "provider"])
RETRIES = Histogram("llm_retry_attempts", "retries per call")

with RETRIES.time():
    # call with retry
    ...
except RateLimitError as e:
    RATE_LIMITS.labels(model=e.model, provider=e.provider).inc()

Alert when the 429 rate exceeds 1% of traffic for a model. That indicates misconfigured limits or a provider incident. Structured logs should include attempt, retry_after, and bucket_remaining so you can reconstruct the event after the fact.

Step 7: Verify with fault injection

To confirm you handle rate limit errors production LLM apps must survive a simulated outage. Stand up a mock that returns 429 for every request, then point your client at it.

# pytest with respx
import respx, httpx, pytest

@respx.mock
def test_handles_429():
    respx.post("https://api.example.com/v1/chat").mock(
        return_value=httpx.Response(429, headers={"retry-after": "0.1"}))
    with pytest.raises(RuntimeError):
        call_with_fallback()  # should exhaust retries and raise

Run a load test at 2x expected peak with the mock returning 429 every tenth call. If p99 latency stays bounded and no unhandled exception escapes, your handling works. For deeper validation, use Toxiproxy to inject latency and 429s at the network layer.

How to verify success in production

After deploy, watch the llm_rate_limits_total metric. Success means: 429s occur but calls eventually succeed; retry histograms show most calls needing 0–1 retry; no customer-facing 500s traced to RateLimitError. If you see cascading failures, tighten the token bucket or reduce concurrency.

Step 8: Set per-tenant quotas

Multi-tenant apps must isolate limits. A single noisy tenant can consume your global bucket. Partition the token bucket by API key or org id.

buckets = defaultdict(lambda: TokenBucket(rate=10, capacity=20))

def call_for_tenant(tenant, msg):
    if not buckets[tenant].consume():
        raise TemporaryUnavailable()
    return call_llm(msg)

This prevents one user from triggering provider-wide 429s that block everyone. Enforce a global bucket on top to respect the provider’s account limit.

Step 9: Graceful degradation

When all models are limited, return a cached response or a static message. Users prefer a stale answer over a spinner.

def get_completion(msg):
    try:
        return call_with_fallback(msg)
    except RuntimeError:
        return cache.get_latest(msg) or "Service busy, try again shortly."

This closes the loop on resilience. You have moved from reacting to 429s to designing around them. The steps above—classify, back off, throttle, fall back, cache, instrument, test, isolate, degrade—are the minimum bar for shipping LLM features that survive contact with real traffic.