Load balancing across LLM providers: a practical guide

Running a single LLM provider in production is a liability the moment you hit a rate limit or a regional outage. Load balancing LLM providers lets you absorb throttling, degraded endpoints, and price swings without rewriting your application logic. This guide gives an ordered path from a naive retry loop to policy-driven routing that survives real agent traffic.

1. Define your routing primitives

Treat every backend as an OpenAI-compatible HTTP endpoint. That single assumption removes most vendor SDK lock-in and makes load balancing LLM providers a networking problem instead of a refactoring project. If you standardize on the /v1/chat/completions shape, you can swap Anthropic, Google, or a self-hosted vLLM node behind the same client.

Define a minimal client wrapper:

from openai import OpenAI

def call_provider(base_url: str, api_key: str, model: str, messages: list):
    client = OpenAI(base_url=base_url, api_key=api_key)
    return client.chat.completions.create(model=model, messages=messages)

Keep the model string namespaced, e.g. openai/gpt-4o or anthropic/claude-3-5-sonnet, so your router knows the backing provider without extra lookups. Avoid embedding provider-specific fields in the request body at this layer; push those into a transformation step later.

Why OpenAI-compatible first

The OpenAI SDK is the lowest common denominator. Every major inference gateway and most model vendors expose an emulation layer. Writing against it means your load balancer only needs to change base_url and api_key per upstream. You lose some native features, but you gain the ability to move traffic in milliseconds.

2. Weighted round-robin with live health

Naive random selection ignores capacity. Assign each provider a weight derived from your quota, observed latency, and error rate. Rotate requests using a simple counter, but skip providers marked unhealthy.

import time
from collections import defaultdict

providers = {
    "openai": {"url": "...", "key": "...", "weight": 5, "healthy": True, "cooldown": 0},
    "anthropic": {"url": "...", "key": "...", "weight": 3, "healthy": True, "cooldown": 0},
}
counter = defaultdict(int)

def pick_provider():
    now = time.time()
    active = [p for p in providers
              if providers[p]["healthy"] or now > providers[p]["cooldown"]]
    if not active:
        raise RuntimeError("no healthy providers")
    total = sum(providers[p]["weight"] for p in active)
    r = (counter["global"] % total)
    counter["global"] += 1
    for p in active:
        r -= providers[p]["weight"]
        if r < 0:
            return p

Health checks should be cheap: a cached completion or a /v1/models fetch every 10–30 seconds. A common pitfall is marking a provider dead on a single 429; that flaps your traffic and amplifies load on the survivor. Use a sliding window of errors—for example, three failures in 20 seconds triggers a 30-second cooldown with jitter.

Dynamic weights matter. If your Anthropic quota is half exhausted, drop its weight rather than waiting for hard errors. Load balancing LLM providers is a continuous control loop, not a static config.

3. Automatic fallback on transient failures

Even solid load balancing LLM providers requires a fallback path. Wrap the call in a retry that catches RateLimitError, APIConnectionError, and timeouts, then shifts to the next provider in a preference list.

from openai import RateLimitError, APIConnectionError
import tenacity

@tenacity.retry(
    retry=tenacity.retry_if_exception_type((RateLimitError, APIConnectionError)),
    stop=tenacity.stop_after_attempt(3),
    retry_error_callback=lambda s: s.args[0]
)
def routed_completion(messages, model_map):
    p = pick_provider()
    try:
        return call_provider(providers[p]["url"], providers[p]["key"], model_map[p], messages)
    except Exception as e:
        providers[p]["healthy"] = False
        providers[p]["cooldown"] = time.time() + 30 + random.uniform(0, 10)
        raise

Tradeoff: fallback changes the model mid-request, so your prompt must not depend on provider-specific behavior like function-calling syntax. Standardize on the OpenAI chat schema and treat tool calls as JSON. If you need strict model parity, restrict fallback to same-family models (e.g., gpt-4o → gpt-4o-mini).

A gateway such as n4n.ai performs automatic fallback when a provider is rate-limited or degraded, but owning this logic locally helps you debug routing decisions and avoid surprise model swaps during evaluations.

Idempotency and retries

LLM calls are not naturally idempotent—resending the same prompt can double-spend tokens. Use a client-generated request_id and have your provider or gateway dedupe within a short window. Without this, a retry storm during a provider outage can 10x your bill.

4. Honor client routing directives and cache hints

Sophisticated callers want to pin a request to a provider or signal caching. Forward x-routing-directive or similar headers, and pass through cache_control fields in the body. Providers like Anthropic use cache_control on system blocks; OpenAI uses user and metadata.

def call_with_directive(p, messages, cache_system: bool):
    client = OpenAI(base_url=providers[p]["url"], api_key=providers[p]["key"])
    if cache_system:
        messages[0]["cache_control"] = {"type": "ephemeral"}
    return client.chat.completions.create(
        model=providers[p]["model"],
        messages=messages,
        extra_headers={"x-routing-directive": f"prefer:{p}"}
    )

Ignoring cache-control throws away free latency and cost wins. When load balancing LLM providers, ensure your router does not strip these fields. A proxy that rewrites the message list must preserve cache_control markers or you will silently lose prefix caching.

Routing directives in practice

Allow an upstream service to say prefer:anthropic for a specific tenant that validated outputs there. Your balancer should treat this as a soft preference: if Anthropic is unhealthy, fall back to the weighted list but log the override. Hard pins (require:openai) should only be used for eval harnesses.

5. Meter per-token usage and enforce budgets

You cannot balance what you do not measure. Every response returns usage.prompt_tokens and usage.completion_tokens. Accumulate per provider and per app tenant.

usage_ledger = defaultdict(lambda: {"prompt": 0, "completion": 0})

def track(p, resp, tenant="default"):
    u = resp.usage
    key = f"{p}:{tenant}"
    usage_ledger[key]["prompt"] += u.prompt_tokens
    usage_ledger[key]["completion"] += u.completion_tokens

Set hard caps: if a provider’s daily token budget hits 80%, drop its weight to zero instead of failing closed. Per-token metering also exposes unbalanced routing—if one provider silently eats 90% of traffic, your health weights are wrong. Export these counters to Prometheus so you can alert on skew.

Multi-tenant apps need per-tenant limits. A noisy neighbor can exhaust your OpenAI quota and starve Anthropic-bound requests. Track at the tenant level and apply weighted shaping:

if usage_ledger[f"openai:{tenant}"]["prompt"] > TENANT_CAP:
    providers["openai"]["weight"] = 0

6. Test with chaos and shadow traffic

Spin up a fault injector that returns 429s for 20% of calls on one provider. Watch whether your fallback actually moves load without stampeding. Shadow traffic—mirroring live prompts to a secondary provider without using the response—catches compatibility bugs early.

Pitfalls:

• Sticky sessions break stateless balancing. Avoid per-user pinning unless latency profiling demands it.
• Model mismatch: claude-3-5-sonnet and gpt-4o are not interchangeable for structured output. Keep a compatibility matrix.
• Timeout tuning: too short and you fallback unnecessarily; too long and p99 suffers. Start at 8s for chat, 30s for long completions.
• Cache poisoning: if you cache responses by prompt hash across providers, you may serve an Anthropic answer to an OpenAI-only feature.

Tradeoff: stronger consistency (same model always) reduces variance but sacrifices availability. Agentic apps usually prefer availability; batch jobs may prefer consistency. Make this a config flag, not a code fork.

Shadow evaluation

Run a nightly job that replays 1% of production prompts to each provider and diffs the JSON schema compliance. This catches silent drift when a provider updates its model. Your router should exclude shadow calls from token budgets.

7. Operational checklist

Before shipping multi-provider routing:

• Provider weights derived from real quota, not guesswork.
• Health cooldown uses jitter to prevent synchronized retries.
• Fallback preserves request schema; no provider-only fields required.
• Cache-control and routing headers forwarded untouched.
• Per-token ledger exported to metrics (Prometheus, etc.).
• Chaos test in staging with forced provider degradation.
• Alert on weight collapse (all traffic on one provider).
• Request-id dedup configured to prevent retry double-spend.

Load balancing LLM providers is not a one-time feature; it is a control loop. The moment you stop watching error budgets and token spreads, a quiet provider outage becomes a full outage. Build the meters first, the router second, and the fallback last. The code above is a starting point—harden it with real quotas and a chaos schedule before trusting it with agent traffic.