Postmortem: recovering from a provider rate limit outage

A rate limit outage recovery postmortem should do more than assign blame. It must produce an actionable path that reduces mean time to detection and mean time to recovery the next time a primary LLM provider returns 429s at scale. This guide walks through the ordered steps we used to survive a throttling event, including the code and configs that actually moved the needle.

Detect the outage before your users do

You cannot recover from a rate limit outage if you learn about it from support tickets. Instrument every LLM call at the token and status level, and treat 429s as a first-class signal.

Token-level metrics

Emit a counter for completions and a histogram for latency, tagged by provider and model. A sudden drop in successful completions coupled with a spike in 429 status codes is your early warning. Do not wait for error rates to cross 50%; throttling degrades quietly.

import prometheus_client as prom

req_total = prom.Counter(
    "llm_requests_total",
    "Total LLM requests",
    ["provider", "model", "status"],
)

def track(provider, model, status):
    req_total.labels(provider=provider, model=model, status=status).inc()

Add a synthetic canary that calls a cheap model every 30 seconds from a separate credential pool. If the canary gets 429s while production does not, you are hitting a tenant-specific quota, not a provider-wide failure.

Distinguish 429 from 5xx

A 429 means “slow down”. A 503 means “I am broken”. Your alerting should page on 429 rate crossing a threshold (e.g., >5% of traffic for 2 minutes) but route it to a different on-call than infrastructure 5xx storms. Mixing them delays the right response.

if resp.status_code == 429:
    track(provider, model, "429")
    # trigger throttling playbook
elif resp.status_code >= 500:
    track(provider, model, "5xx")
    # trigger provider degradation playbook

Pitfall: aggregating all non-200s into one bucket hides the fact that you are being rate limited rather than experiencing a hard outage. We once spent 20 minutes scaling pods because Grafana showed “errors” without the split.

Triage: stop the bleeding

Once confirmed, your first goal is to protect the majority of requests while you investigate. You have two levers: shed load or queue it, and shift traffic.

Shed load or queue

If the workload is asynchronous (summarization, batch labeling), push pending jobs into a bounded queue and process at a rate the provider can sustain. For synchronous user-facing calls, return a structured retry-after response to the client instead of blocking your worker pool.

from redis import Redis
from redis.queue import Queue

q = Queue("llm_jobs", connection=Redis())

def enqueue(job_payload):
    if q.len() > 10_000:
        raise OverflowError("queue full, reject with 429")
    q.enqueue(process_job, job_payload)

Tradeoff: queueing improves resilience but increases tail latency. Set a max queue depth and reject beyond it. A rejected job is better than a worker deadlock at 3 a.m.

Enable fallback routing

If you front your providers with a gateway such as n4n.ai, automatic fallback when a provider is rate-limited or degraded can shift traffic without code changes. With an OpenAI-compatible endpoint, you can also pass a routing directive header to force a secondary provider.

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "x-n4n-route: provider=secondary" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'

If you roll your own fallback, implement a weighted shuffle of providers and short-circuit on 429. Do not fail open to the most expensive model because it is the only one not limited.

providers = ["primary", "secondary", "tertiary"]

def call_with_fallback(messages):
    for prov in providers:
        try:
            return do_completion(prov, messages)
        except RateLimitError:
            continue
    raise AllProvidersThrottled()

Common mistake: forgetting to cap concurrent in-flight requests to the fallback. You can DDOS your secondary provider and cascade the outage.

Root cause analysis without guessing

A rate limit outage recovery postmortem demands a cause, not a guess. Provider status pages often show “all systems operational” during soft throttling.

Pull provider headers

Most providers return x-ratelimit-remaining, x-ratelimit-reset, and retry-after. Log these on every 429. If remaining is 0 but reset is seconds away, you hit a hard quota. If retry-after is large (minutes), you triggered a concurrent request cap.

if resp.status_code == 429:
    reset = resp.headers.get("x-ratelimit-reset")
    remaining = resp.headers.get("x-ratelimit-remaining")
    log.warning("throttled", reset=reset, remaining=remaining)

Reproduce with constrained concurrency

Write a small load test that ramps concurrency to the documented limit minus a buffer. If you reproduce 429s below the documented cap, the provider changed limits silently—a known industry pattern. Do this from a staging account to avoid making production worse.

hey -n 1000 -c 20 -m POST \
  -H "Authorization: Bearer $STAGING_KEY" \
  -d '{"model":"mini","messages":[]}' \
  https://api.provider.com/v1/chat/completions

Mitigate: short-term fixes

Backoff with jitter

Naive fixed-delay retries create retry storms. Use exponential backoff with full jitter and a max attempts cap. After 5 failures, drop to queue instead of retrying inline.

import random, time

def backoff(attempt):
    base = 0.5
    cap = 30
    sleep = min(cap, base * 2 ** attempt)
    time.sleep(sleep * random.random())

Cache aggressively

Forward provider cache-control hints and reuse completions for deterministic prompts. An OpenAI-compatible gateway will pass through cache-control if you set it. This cuts both token spend and 429 probability.

{
  "model": "gpt-4o-mini",
  "messages": [{"role": "user", "content": "def add(a,b):"}],
  "cache_control": {"type": "ephemeral"}
}

Tradeoff: caching reduces spend and 429s but can serve stale logic for code-gen tasks. Set TTLs per task type. Never cache personalized or time-sensitive outputs.

Long-term hardening

The goal of any rate limit outage recovery postmortem is to ensure the next event is a non-incident.

Multi-provider routing directives

Encode routing policy in client headers, not in deployed config. That lets you shift traffic during an outage with a single flag flip rather than a risky deploy.

{
  "x-n4n-route": "provider=secondary; fallback=tertiary"
}

Per-token metering to spot abuse

Track per-tenant token usage. A single tenant can trigger a shared provider quota, throttling everyone. Per-token usage metering exposes this quickly and lets you apply tenant-level RPM guards.

def meter(tenant, tokens):
    usage.labels(tenant=tenant).inc(tokens)

Capacity planning

Negotiate higher limits with providers based on historical p95 concurrency. Keep a spreadsheet of documented vs observed caps; update it quarterly. Shadow-test new providers with mirrored traffic at low percentage before promoting them to fallback tier.

Common pitfalls

Ignoring retry storms

A 429 with a 1s retry-after and 50 workers becomes 50 requests per second of pure noise. Cap concurrent retries globally with a semaphore.

from threading import Semaphore
retry_sem = Semaphore(5)

def safe_retry(fn):
    with retry_sem:
        return fn()

Treating all 429s equally

Some 429s are per-minute token caps; others are per-second request caps. Your backoff must differ. Inspect headers and branch.

Missing client-side timeouts

If your client hangs for 60s waiting on a throttled provider, your users see a spinny wheel even after you fail over. Set 10s connect, 30s read timeouts.

requests.post(url, json=payload, timeout=(10, 30))

Writing a postmortem with no owners

A rate limit outage recovery postmortem only matters if the action items close. Assign owners and due dates in the same doc, and link them to sprint tickets.

Postmortem template

Keep it short. Use this structure:

## Incident: <date> Provider X 429 storm
- Detection: 03:12 UTC via 429 alert
- Impact: 18% of completions failed for 22 min
- Root cause: Tenant A batch job exceeded shared RPM limit
- Mitigation: Queue depth cap + fallback to secondary
- Action items:
  - [ ] Per-tenant RPM guardrails (owner: @infra, due: +1wk)
  - [ ] Header-based routing in prod (owner: @api, due: +2wk)
  - [ ] Jitter backoff in SDK (owner: @client, due: +3d)

The next 429 wave will come. The difference between a 2 a.m. page and a self-healing graph is whether you executed the path above before the provider silently lowered your quota.