Rate limits are the single most common cause of production failures in LLM applications. When your traffic spikes or a provider degrades, a single endpoint becomes a single point of failure. Load balancing rate limit errors requires distributing requests across multiple providers and models so that one provider’s quota exhaustion doesn’t cascade into user-facing errors. This guide walks through the strategies, implementation patterns, and operational realities of building resilient LLM inference routing.
Why naive retries make rate limits worse
Most teams start with exponential backoff and retry logic. This works for transient network errors but fails catastrophically against rate limits. When a provider returns 429, the retry-after header often suggests waiting seconds or minutes. If you have ten concurrent requests and all hit the limit, backing off synchronously creates a thundering herd when the window resets — every client retries at once, triggering another wave of 429s.
The math is simple: if your application makes 100 requests/minute and the provider allows 60, you will hit limits. Retries don’t create capacity; they just shift when the failure occurs. Load balancing rate limit errors means adding actual capacity by routing to providers with available quota.
# Naive retry — don't do this in production
import time
import openai
def naive_retry(client, messages, max_retries=3):
for attempt in range(max_retries):
try:
return client.chat.completions.create(
model="gpt-4o",
messages=messages
)
except openai.RateLimitError as e:
wait = min(2 ** attempt * 5, 60) # exponential backoff
time.sleep(wait)
raise Exception("Exhausted retries")Strategy 1: Round-robin across providers with quota awareness
The simplest effective approach distributes requests across multiple provider accounts or API keys. Each provider maintains independent rate limit buckets. If you have three OpenAI organization accounts each allowing 500 RPM, round-robin gives you 1500 RPM aggregate — but only if you track per-provider usage and stop sending to exhausted accounts.
import threading
from dataclasses import dataclass
from typing import Optional
import openai
@dataclass
class ProviderSlot:
client: openai.OpenAI
rpm_limit: int
requests_this_minute: int = 0
window_start: float = 0
lock: threading.Lock = threading.Lock()
class QuotaAwareBalancer:
def __init__(self, slots: list[ProviderSlot]):
self.slots = slots
self._index = 0
self._index_lock = threading.Lock()
def _can_use(self, slot: ProviderSlot) -> bool:
now = time.time()
with slot.lock:
if now - slot.window_start >= 60:
slot.requests_this_minute = 0
slot.window_start = now
return slot.requests_this_minute < slot.rpm_limit
def _increment(self, slot: ProviderSlot):
with slot.lock:
slot.requests_this_minute += 1
def get_available_slot(self) -> Optional[ProviderSlot]:
# Try each slot once, starting from last used
with self._index_lock:
start = self._index
for i in range(len(self.slots)):
idx = (start + i) % len(self.slots)
slot = self.slots[idx]
if self._can_use(slot):
with self._index_lock:
self._index = (idx + 1) % len(self.slots)
self._increment(slot)
return slot
return NoneThis works but has a blind spot: it assumes all providers have identical latency and error profiles. In practice, Anthropic, Google, and Azure OpenAI have different rate limit structures (TPM vs RPM), different retry-after behaviors, and different model availability.
Strategy 2: Model-aware routing with fallback chains
Production systems need to route based on model capability, not just provider quota. A request for claude-3.5-sonnet shouldn’t fall back to gpt-3.5-turbo silently — the output quality differs. Instead, define explicit fallback chains per capability tier:
from enum import Enum
from typing import Literal
class CapabilityTier(Enum):
REASONING = "reasoning" # complex logic, coding, analysis
GENERAL = "general" # chat, summarization, extraction
FAST = "fast" # classification, routing, simple QA
FALLBACK_CHAINS = {
CapabilityTier.REASONING: [
("anthropic", "claude-3.5-sonnet"),
("openai", "gpt-4o"),
("google", "gemini-1.5-pro"),
],
CapabilityTier.GENERAL: [
("openai", "gpt-4o-mini"),
("anthropic", "claude-3.5-haiku"),
("google", "gemini-1.5-flash"),
],
CapabilityTier.FAST: [
("openai", "gpt-4o-mini"),
("google", "gemini-1.5-flash-8b"),
("anthropic", "claude-3.5-haiku"),
],
}
class TieredRouter:
def __init__(self, clients: dict[str, openai.OpenAI], balancers: dict[str, QuotaAwareBalancer]):
self.clients = clients
self.balancers = balancers # keyed by provider name
def route(self, tier: CapabilityTier, messages: list[dict], **kwargs):
last_error = None
for provider, model in FALLBACK_CHAINS[tier]:
balancer = self.balancers.get(provider)
if not balancer:
continue
slot = balancer.get_available_slot()
if not slot:
continue # provider exhausted, try next in chain
client = self.clients[provider]
try:
# Normalize model name per provider
response = client.chat.completions.create(
model=model,
messages=messages,
**kwargs
)
# Attach routing metadata for observability
response._routing = {"provider": provider, "model": model, "tier": tier.value}
return response
except Exception as e:
last_error = e
# Don't retry same provider — move to next in chain
continue
raise Exception(f"All providers exhausted for {tier.value}: {last_error}")Tradeoff: Fallback chains increase latency variance. A request that hits the third provider in the chain adds two failed connection attempts. Mitigate this by keeping chains short (2-3 providers) and using async health checks to deprioritize degraded providers before they cause failures.
Strategy 3: Token-aware load balancing
Rate limits are often token-based (TPM) not request-based (RPM). A single 100k-token request consumes more quota than ten 1k-token requests. If you only track request counts, you’ll under-utilize providers with high token limits or accidentally exceed token quotas on providers with low ones.
@dataclass
class TokenBucket:
tpm_limit: int
tokens_used: int = 0
window_start: float = 0
lock: threading.Lock = threading.Lock()
def try_consume(self, estimated_tokens: int) -> bool:
now = time.time()
with self.lock:
if now - self.window_start >= 60:
self.tokens_used = 0
self.window_start = now
if self.tokens_used + estimated_tokens <= self.tpm_limit:
self.tokens_used += estimated_tokens
return True
return False
def refund(self, actual_tokens: int, estimated_tokens: int):
# Adjust for estimation error after response
with self.lock:
self.tokens_used += actual_tokens - estimated_tokensEstimate tokens before sending using the provider’s tokenizer (or a fast approximation like tiktoken). Refund the difference after the response arrives. This prevents a single large request from blocking smaller ones that could have succeeded.
Pitfall: Token estimation is imperfect. Providers count tokens differently (some include system prompts, some don’t). Build in a 10-15% safety margin on estimates, and always handle the case where actual usage exceeds the estimate.
Strategy 4: Priority queues for mixed workloads
Not all requests are equal. Background batch jobs can wait; user-facing chat cannot. Implement priority queuing so that high-priority traffic gets first access to quota, while lower-priority work fills gaps.
import heapq
from dataclasses import dataclass, field
from enum import IntEnum
class Priority(IntEnum):
INTERACTIVE = 0 # user-facing, latency-sensitive
BACKGROUND = 1 # batch, async, can retry later
BEST_EFFORT = 2 # analytics, logging, non-critical
@dataclass(order=True)
class QueuedRequest:
priority: Priority
timestamp: float = field(compare=True)
tier: CapabilityTier = field(compare=False)
messages: list = field(compare=False)
kwargs: dict = field(compare=False)
future: asyncio.Future = field(compare=False)
class PriorityRouter:
def __init__(self, router: TieredRouter, max_queue_size: int = 1000):
self.router = router
self.queue: list[QueuedRequest] = []
self.queue_lock = asyncio.Lock()
self.max_queue_size = max_queue_size
self._worker_task = None
async def enqueue(self, priority: Priority, tier: CapabilityTier, messages: list, **kwargs):
if len(self.queue) >= self.max_queue_size:
raise Exception("Queue full")
future = asyncio.get_event_loop().create_future()
req = QueuedRequest(priority, time.time(), tier, messages, kwargs, future)
async with self.queue_lock:
heapq.heappush(self.queue, req)
return await future
async def _worker(self):
while True:
async with self.queue_lock:
if not self.queue:
await asyncio.sleep(0.1)
continue
req = heapq.heappop(self.queue)
try:
# Run sync router in thread pool
loop = asyncio.get_event_loop()
response = await loop.run_in_executor(
None, self.router.route, req.tier, req.messages, **req.kwargs
)
req.future.set_result(response)
except Exception as e:
req.future.set_exception(e)This adds operational complexity: you need queue depth monitoring, priority inversion detection, and backpressure signals to upstream callers. Only add it when you have measurable contention between workload types.
Handling provider-specific nuances
Each provider exposes rate limits differently. Your balancer must normalize these into a common internal model.
| Provider | Limit type | Header / response | Notable behavior |
|---|---|---|---|
| OpenAI | RPM + TPM | x-ratelimit-limit-requests, x-ratelimit-limit-tokens |
Separate buckets per model |
| Anthropic | RPM + TPM | anthropic-ratelimit-requests-limit, anthropic-ratelimit-tokens-limit |
Shared across models in tier |
| Google Vertex | RPM | Quota project-level | Requires GCP quota increases |
| Azure OpenAI | TPM + RPM | Deployment-level | Per-deployment limits, not subscription |
Parse rate limit headers on every response and update your local bucket state. This lets you react faster than waiting for 429s.
def update_from_headers(self, provider: str, headers: dict):
if provider == "openai":
rpm_remaining = int(headers.get("x-ratelimit-remaining-requests", 0))
tpm_remaining = int(headers.get("x-ratelimit-remaining-tokens", 0))
# Update local buckets...
elif provider == "anthropic":
rpm_remaining = int(headers.get("anthropic-ratelimit-requests-remaining", 0))
tpm_remaining = int(headers.get("anthropic-ratelimit-tokens-remaining", 0))
# ...Critical: Some providers (notably Azure) don’t return remaining quota on successful responses — only on 429. For these, you must maintain a conservative local estimate and accept occasional 429s as the signal to recalibrate.
Observability: what to measure
You cannot tune what you don’t measure. At minimum, emit these metrics per provider and model:
# Prometheus-style metrics
from prometheus_client import Counter, Histogram, Gauge
requests_total = Counter(
"llm_requests_total",
"Total requests",
["provider", "model", "tier", "status"] # status: success, rate_limited, error
)
request_latency = Histogram(
"llm_request_latency_seconds",
"End-to-end latency",
["provider", "model", "tier"]
)
quota_utilization = Gauge(
"llm_quota_utilization_ratio",
"Current usage / limit",
["provider", "limit_type"] # rpm, tpm
)
fallback_depth = Histogram(
"llm_fallback_depth",
"Number of providers tried before success",
["tier"]
)
queue_depth = Gauge(
"llm_queue_depth",
"Pending requests by priority",
["priority"]
)Alert on:
quota_utilization > 0.85sustained for 5 minutes (proactive scaling signal)fallback_depth > 1increasing (primary provider degradation)queue_depthgrowing on INTERACTIVE priority (capacity shortage)
Common pitfalls
1. Ignoring model compatibility. Falling back from gpt-4o to gpt-3.5-turbo changes output structure, reasoning ability, and context window. Define compatibility matrices, not just provider lists.
2. Caching routing decisions. A provider that was healthy 30 seconds ago may be rate-limited now. Re-evaluate provider availability on every request, or at minimum every few seconds with a background health check.
3. Single-threaded quota tracking. The QuotaAwareBalancer above uses locks. At high QPS, lock contention becomes a bottleneck. Use lock-free atomic counters or shard buckets by worker thread.
4. No circuit breaker. If a provider returns 5xx errors, stop routing to it for a cooldown period. Continuing to send traffic wastes quota on other providers (due to retries) and delays user requests.
class CircuitBreaker:
def __init__(self, failure_threshold=5, cooldown_seconds=30):
self.failure_threshold = failure_threshold
self.cooldown_seconds = cooldown_seconds
self.failures = 0
self.last_failure = 0
self.lock = threading.Lock()
def record_success(self):
with self.lock:
self.failures = 0
def record_failure(self):
with self.lock:
self.failures += 1
self.last_failure = time.time()
def is_open(self) -> bool:
with self.lock:
if self.failures >= self.failure_threshold:
if time.time() - self.last_failure < self.cooldown_seconds:
return True
# Half-open: allow one request to test
self.failures = self.failure_threshold - 1
return False5. Forgetting streaming responses. Streaming holds connections open longer, consuming connection pool slots and potentially hitting concurrent request limits even when RPM/TPM are fine. Track active streams separately.
Putting it together: a minimal production router
The patterns above compose into a router that handles the common cases without over-engineering:
class ProductionRouter:
def __init__(self, config: dict):
self.clients = self._build_clients(config)
self.balancers = self._build_balancers(config)
self.circuit_breakers = {
provider: CircuitBreaker() for provider in config["providers"]
}
self.tiered_router = TieredRouter(self.clients, self.balancers)
def route(self, tier: CapabilityTier, messages: list[dict], **kwargs):
for provider, model in FALLBACK_CHAINS[tier]:
if self.circuit_breakers[provider].is_open():
continue
balancer = self.balancers[provider]
slot = balancer.get_available_slot()
if not slot:
continue
client = self.clients[provider]
try:
response = client.chat.completions.create(
model=model, messages=messages, **kwargs
)
self.circuit_breakers[provider].record_success()
# Update quota from headers
self._update_quota(provider, response.headers)
return response
except Exception as e:
self.circuit_breakers[provider].record_failure()
if isinstance(e, RateLimitError):
balancer.mark_exhausted(slot)
continue
raise Exception("All providers exhausted")This is ~200 lines of core logic. The rest is configuration, observability, and deployment plumbing.
When to use a gateway instead of building this
If you’re managing more than 3-4 providers, or your team doesn’t want to own the operational burden of quota tracking, header parsing, and fallback logic, an inference gateway handles this at the infrastructure layer. n4n.ai exposes a single OpenAI-compatible endpoint that addresses 240+ models, performs automatic fallback when a provider is rate-limited or degraded, meters per-token usage, and forwards provider cache-control hints — so your application code stays focused on product logic, not routing logic.
The gateway approach trades control for operational simplicity. You lose the ability to implement custom priority queues or token-aware scheduling, but you gain immediate resilience without maintaining provider-specific integrations.
Summary checklist
- Track per-provider RPM and TPM quotas locally, updated from response headers
- Define explicit fallback chains by capability tier, not by provider
- Estimate tokens pre-request; refund/adjust post-response
- Implement circuit breakers per provider for 5xx errors
- Emit quota utilization, fallback depth, and latency metrics
- Alert at 85% quota utilization, not at 100%
- Test fallback behavior under load before production traffic hits it
Rate limits are a capacity problem, not a retry problem. The only durable fix is more quota — either by purchasing more from one provider, or by architecting your system to use multiple providers simultaneously. Load balancing rate limit errors across providers turns a hard limit into a soft, manageable constraint.