What is a 429 error and how to handle it gracefully

A 429 error is an HTTP status code that means “too many requests” — the server is telling you that you have exceeded a rate limit or quota within a given time window. When you hit a 429, the request was not processed; you must wait and retry. Understanding what is a 429 error and how to handle it gracefully separates fragile prototypes from production-grade LLM integrations.

How 429 works in practice

Rate limits exist at multiple layers. A provider may enforce limits per API key, per organization, per IP, per model, or per endpoint. Limits are typically expressed as requests per minute (RPM), tokens per minute (TPM), or concurrent requests. Some providers also impose daily or monthly quotas that return 429 once exhausted.

The response usually includes headers that tell you what happened and when to retry:

HTTP/1.1 429 Too Many Requests
Retry-After: 45
X-RateLimit-Limit: 3000
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1725123600

• Retry-After — seconds (or an HTTP date) until you should send the next request.
• X-RateLimit-Limit — the ceiling for the current window.
• X-RateLimit-Remaining — how many requests you have left (zero in this case).
• X-RateLimit-Reset — Unix timestamp when the window resets.

Not every provider returns all of these. Some return a JSON body with a retry_after_ms field. Others return nothing useful at all. Your client must handle missing or malformed headers without crashing.

Why it matters for LLM workloads

LLM inference is expensive and capacity-constrained. Providers protect their GPUs with aggressive limits. A single 429 can cascade: your retry logic hammers the endpoint, you get rate-limited again, latency spikes, and upstream timeouts trigger more retries. This is how a minor quota breach becomes a full outage.

Three characteristics make LLM traffic especially prone to 429 storms:

1. Bursty traffic — batch jobs, eval runs, and user-facing chat all spike at once.
2. Variable token counts — a 200-token request and a 32k-token request consume vastly different quota but often count equally against RPM limits.
3. Provider diversity — each model provider (OpenAI, Anthropic, Google, Cohere, together.ai, Fireworks, etc.) has different limit structures, header conventions, and backoff expectations.

If you route traffic through a gateway that sits in front of multiple providers — such as n4n.ai — you gain a single OpenAI-compatible endpoint with automatic fallback when one provider returns 429 or degrades. The gateway honors your routing directives and forwards provider cache-control hints, but you still need client-side discipline.

Concrete example: resilient retry wrapper

Below is a minimal, production-style async wrapper in Python. It respects Retry-After, implements exponential backoff with jitter, caps total retry time, and surfaces a typed exception when the budget is exhausted.

import asyncio
import random
import time
from dataclasses import dataclass
from typing import Optional
import httpx

@dataclass
class RateLimitExceeded(Exception):
    retry_after: Optional[float] = None
    limit: Optional[int] = None
    reset_at: Optional[float] = None

async def chat_completion_with_backoff(
    client: httpx.AsyncClient,
    url: str,
    payload: dict,
    *,
    max_retries: int = 5,
    base_delay: float = 1.0,
    max_delay: float = 60.0,
    total_timeout: float = 120.0,
) -> dict:
    """
    POST with 429-aware retry. Returns parsed JSON on success.
    Raises RateLimitExceeded if retries exhausted or total_timeout exceeded.
    """
    deadline = time.monotonic() + total_timeout
    attempt = 0

    while True:
        if time.monotonic() > deadline:
            raise RateLimitExceeded(retry_after=None)

        resp = await client.post(url, json=payload, timeout=30.0)

        if resp.status_code != 429:
            resp.raise_for_status()
            return resp.json()

        # Parse Retry-After (seconds or HTTP-date)
        retry_after = _parse_retry_after(resp.headers.get("Retry-After"))
        limit = _parse_int(resp.headers.get("X-RateLimit-Limit"))
        reset_at = _parse_float(resp.headers.get("X-RateLimit-Reset"))

        # If server gave explicit guidance, prefer it; otherwise exponential backoff
        if retry_after is not None:
            delay = min(retry_after, max_delay)
        else:
            delay = min(base_delay * (2 ** attempt) + random.uniform(0, 0.5), max_delay)

        # Don't sleep past our total budget
        remaining = deadline - time.monotonic()
        if delay > remaining:
            raise RateLimitExceeded(retry_after=retry_after, limit=limit, reset_at=reset_at)

        await asyncio.sleep(delay)
        attempt += 1

        if attempt >= max_retries:
            raise RateLimitExceeded(retry_after=retry_after, limit=limit, reset_at=reset_at)


def _parse_retry_after(value: Optional[str]) -> Optional[float]:
    if value is None:
        return None
    try:
        return float(value)
    except ValueError:
        # Could be an HTTP-date; parse if needed
        return None


def _parse_int(value: Optional[str]) -> Optional[int]:
    try:
        return int(value) if value is not None else None
    except ValueError:
        return None


def _parse_float(value: Optional[str]) -> Optional[float]:
    try:
        return float(value) if value is not None else None
    except ValueError:
        return None

Key points in this implementation:

• Total timeout budget — prevents a single call from blocking a worker indefinitely.
• Jitter — random.uniform(0, 0.5) breaks synchronization across multiple clients retrying simultaneously.
• Header precedence — explicit Retry-After beats calculated backoff.
• Typed exception — callers can inspect retry_after, limit, reset_at for logging, metrics, or user-facing messaging.

Client-side rate limiting: stop the 429 before it happens

Retry logic is a safety net. The real fix is client-side rate limiting that keeps you under the provider’s ceiling. Two patterns work well:

Token bucket (smooths bursts)

import asyncio
import time
from dataclasses import dataclass, field

@dataclass
class TokenBucket:
    rate: float          # tokens per second
    capacity: int        # max burst
    tokens: float = field(init=False)
    last: float = field(init=False)
    _lock: asyncio.Lock = field(default_factory=asyncio.Lock, init=False)

    def __post_init__(self):
        self.tokens = float(self.capacity)
        self.last = time.monotonic()

    async def take(self, cost: int = 1) -> None:
        async with self._lock:
            while self.tokens < cost:
                now = time.monotonic()
                since_last = now - self.last
                self.tokens = min(self.capacity, self.tokens + since_last * self.rate)
                self.last = now
                if self.tokens < cost:
                    # Sleep until we have enough tokens
                    needed = cost - self.tokens
                    await asyncio.sleep(needed / self.rate)
            self.tokens -= cost

Initialize one bucket per provider/key with the documented RPM/TPM limits. Call await bucket.take(tokens_estimated) before each request. This eliminates most 429s entirely.

Fixed window with Redis (distributed systems)

If you run multiple workers, a local bucket isn’t enough. Use a Redis-backed sliding window:

-- rate_limit.lua
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window_ms = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])

redis.call('ZREMRANGEBYSCORE', key, 0, now - window_ms)
local current = redis.call('ZCARD', key)

if current + cost > limit then
    local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
    local retry_after = 0
    if #oldest > 0 then
        retry_after = math.ceil((oldest[2] + window_ms - now) / 1000)
    end
    return {0, retry_after, limit, current}
end

redis.call('ZADD', key, now, now .. '-' .. math.random())
redis.call('PEXPIRE', key, window_ms)
return {1, 0, limit, current + cost}

Call from Python:

async def acquire_slot(redis, key: str, limit: int, window_ms: int, cost: int = 1) -> tuple[bool, int]:
    now = int(time.time() * 1000)
    allowed, retry_after, limit, current = await redis.evalsha(
        SHA, 1, key, limit, window_ms, now, cost
    )
    return bool(allowed), retry_after

This gives you precise, distributed enforcement with a single round-trip.

Common misconceptions

“429 Means the model is down”

False. 429 means you exceeded your allocation. The model is likely healthy. Treat it as a quota signal, not a health signal. If you want health checks, probe a lightweight endpoint (e.g., /models) separately.

“I should retry immediately with a different key”

Rotating keys to bypass limits violates most providers’ terms of service and gets your accounts banned. If you need higher throughput, request a limit increase or use a gateway that load-balances across legitimate accounts.

“Exponential backoff alone is enough”

Backoff without a total timeout budget causes request pile-up. Workers accumulate in retry loops, memory grows, and the next traffic spike OOMs the process. Always cap total retry time.

“The Retry-After header is always seconds”

The spec allows an HTTP-date (e.g., Retry-After: Wed, 21 Oct 2015 07:28:00 GMT). Parse both formats. Many LLM providers only send seconds, but a robust client handles both.

“Rate limits are static”

Providers change limits without notice. Some adjust dynamically based on cluster load. Log every 429 with the response headers; alert on sustained rate-limit hit rates. Treat limits as runtime configuration, not constants.

Observability you actually need

Add these metrics to your dashboard:

Alert when rate(llm_requests_total{status="429"}[5m]) > 0.05 * rate(llm_requests_total[5m]) — more than 5% of requests hitting 429 indicates misconfigured client limits or a provider-side change.

Putting it together

A resilient LLM client stack has three layers:

1. Client-side limiter (token bucket or Redis window) — prevents 429s.
2. Retry wrapper — handles the inevitable 429s with backoff, jitter, and a hard timeout.
3. Observability — metrics and alerts that tell you when limits drift or traffic patterns shift.

Skip any layer and you will eventually get burned. The 429 is not an error to “fix” — it is a protocol signal to respect. Build your client to listen.