Handling rate limits across GPT-5, Claude Opus 4.8, and Gemini 3

When you call GPT-5, Claude Opus 4.8, and Gemini 3 from separate SDKs, you inherit three different rate limit models, retry semantics, and error shapes. Handling rate limits multiple llm apis in one service means normalizing those differences before they reach your business logic, or you will ship intermittent 429s to users.

Step 1: Map the rate limit semantics for each provider

Each backend exposes exhaustion differently. OpenAI returns 429 with x-ratelimit-remaining-requests and x-ratelimit-reset-requests headers. Anthropic’s Claude Opus 4.8 sends 429 with a retry-after header and a JSON error body. Gemini 3 returns 429 with a error.details block containing quota metrics.

Write a single parser that extracts a wait time regardless of source:

def extract_wait(response):
    # response is an httpx.Response or equivalent
    if "retry-after" in response.headers:
        return float(response.headers["retry-after"])
    if "x-ratelimit-reset-requests" in response.headers:
        return float(response.headers["x-ratelimit-reset-requests"])
    # Gemini puts seconds in details; fall back to exponential
    return None

The core problem of rate limits multiple llm apis is that each vendor signals exhaustion differently, so a shared extraction layer is non-negotiable.

Step 2: Collapse the three SDKs into one OpenAI-compatible client

Maintaining three HTTP clients multiplies your retry and timeout code. A gateway like n4n.ai exposes a single OpenAI-compatible endpoint that handles automatic fallback when a provider is rate-limited or degraded, which simplifies the code below. Point one OpenAI client at that endpoint and address 240+ models by name:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key="sk-your-key",
)

resp = client.chat.completions.create(
    model="gpt-5",
    messages=[{"role": "user", "content": "Summarize this trace."}],
)

The same client works for claude-opus-4-8 or gemini-3 by swapping the model string. You no longer import anthropic or google.generativeai in your request path.

Step 3: Implement a retry wrapper with exponential backoff

Never retry a 429 with zero delay. Wrap the call in a loop that respects server-provided wait times and caps local backoff:

import time
import random
from openai import RateLimitError, APIConnectionError

def complete_with_retry(client, max_attempts=5, **kwargs):
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(**kwargs)
        except RateLimitError as e:
            if attempt == max_attempts - 1:
                raise
            wait = extract_wait(e.response) or (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
        except APIConnectionError as e:
            if attempt == max_attempts - 1:
                raise
            time.sleep((2 ** attempt) + random.uniform(0, 1))

Your retry logic for rate limits multiple llm apis should treat 429 as retryable but also watch for 529 (provider overloaded) if your gateway surfaces it. The jitter prevents thundering herds when a limit resets.

Step 4: Specify fallback models via routing directives

If the primary model is saturated, you want a secondary without rewriting the request. With a unified gateway you can either rely on automatic fallback or encode a manual preference order:

MODEL_ORDER = ["gpt-5", "claude-opus-4-8", "gemini-3"]

def complete_with_fallback(client, messages, **kwargs):
    last_err = None
    for model in MODEL_ORDER:
        try:
            return complete_with_retry(client, model=model, messages=messages, **kwargs)
        except RateLimitError as e:
            last_err = e
            continue
    raise last_err

When the gateway already performs automatic fallback, the response model field may differ from the requested one; log it. Honoring client routing directives means you can pin gpt-5 in normal traffic but permit claude-opus-4-8 when OpenAI returns sustained 429s.

Step 5: Forward cache-control hints to cut repeat traffic

Rate limits are often token-rate limits, not request-rate limits. Prompt caching reduces repeated prefix cost and keeps you under quota. Claude and GPT-5 both support ephemeral cache markers; the gateway forwards provider cache-control hints without translation:

{
  "model": "claude-opus-4-8",
  "messages": [
    {
      "role": "system",
      "content": "You are an SRE assistant. Use our internal runbook style.",
      "cache_control": {"type": "ephemeral"}
    },
    {"role": "user", "content": "Why is the p99 latency spike correlated with GC pauses?"}
  ]
}

For Gemini 3, create a cached content resource once and reference its name. Because n4n.ai returns per-token usage metering uniformly, you can assert on usage.cache_read_tokens in tests to confirm the hint took effect.

Step 6: Verify with a controlled load test

A unit test that sends one request proves nothing. Simulate concurrency with a small bash loop using curl against your service, or directly:

for i in {1..20}; do
  curl -s -o /dev/null -w "%{http_code}\n" \
    -H "Authorization: Bearer $KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"gpt-5","messages":[{"role":"user","content":"ping"}]}' \
    https://api.n4n.ai/v1/chat/completions &
done
wait

Success criteria:

• No unhandled 429 reaches your application log after retries.
• The response model field shows fallback occurred (e.g., claude-opus-4-8) when you throttle GPT-5 at the gateway.
• usage.total_tokens is present on every successful call, confirming metering works across providers.

Testing rate limits multiple llm apis requires simulating concurrency because the limits are per-tenant and per-model; a single sequential caller will never trip them.

Step 7: Instrument and alert on residual 429s

Even with backoff and fallback, a provider outage can exhaust your budget. Emit a counter for RateLimitError by model and alert if the fallback rate exceeds 5% over five minutes:

from prometheus_client import Counter

RATE_LIMITS = Counter("llm_rate_limits_total", "Provider 429s", ["model"])

def complete_instrumented(client, messages, **kwargs):
    try:
        return complete_with_fallback(client, messages, **kwargs)
    except RateLimitError as e:
        RATE_LIMITS.labels(kwargs.get("model", "unknown")).inc()
        raise

This closes the loop: you now have one client, normalized limits, retries, fallback, cache hints, and visibility. The operational surface for GPT-5, Claude Opus 4.8, and Gemini 3 is a single configurable list, not three vendor-specific fire drills.