A provider fallback chain is an ordered list of models and providers that your gateway tries sequentially when the primary choice fails, times out, or exceeds your latency budget. Building one that behaves predictably under load requires more than a simple retry loop — you need explicit health signals, per-hop latency budgets, and a policy for what “failure” actually means. This guide walks through the components you need, the decisions you’ll make, and the failure modes that bite teams in production.
Define what triggers a fallback
The first decision is semantic: what counts as a failure worth falling back from? Most teams start with HTTP 5xx and network errors, but that’s insufficient. You also need to handle:
- HTTP 429 (rate limited) — but only if the
Retry-Afterheader exceeds your latency budget - HTTP 400/401/403 — these are not fallback triggers; they indicate a bug in your request or credentials
- Partial success — the provider returns 200 but the response is truncated, malformed, or fails schema validation
- Latency SLO breach — the request succeeds but takes longer than your p99 target
Encode this as a predicate, not scattered if statements:
# fallback_policy.py
from dataclasses import dataclass
from enum import Enum
from typing import Protocol
class FallbackReason(Enum):
NETWORK_ERROR = "network_error"
HTTP_5XX = "http_5xx"
RATE_LIMITED = "rate_limited"
LATENCY_BUDGET_EXCEEDED = "latency_budget_exceeded"
INVALID_RESPONSE = "invalid_response"
@dataclass(frozen=True)
class FallbackPolicy:
max_latency_ms: int
retryable_status_codes: frozenset[int] = frozenset({429, 500, 502, 503, 504})
non_retryable_status_codes: frozenset[int] = frozenset({400, 401, 403, 404, 422})
def should_fallback(self, response: "ProviderResponse", elapsed_ms: int) -> FallbackReason | None:
if response.error is not None:
return FallbackReason.NETWORK_ERROR
if response.status in self.non_retryable_status_codes:
return None # surface to caller immediately
if response.status in self.retryable_status_codes:
if response.status == 429 and response.retry_after_ms <= self.max_latency_ms:
return None # short retry-after, let the caller retry
return FallbackReason.RATE_LIMITED if response.status == 429 else FallbackReason.HTTP_5XX
if elapsed_ms > self.max_latency_ms:
return FallbackReason.LATENCY_BUDGET_EXCEEDED
if not response.is_valid():
return FallbackReason.INVALID_RESPONSE
return NoneStructure the chain as data, not code
Hardcoding if provider == "openai": try_anthropic() creates a maintenance nightmare. Represent each hop as a data structure that your router interprets generically:
{
"fallback_chain": [
{
"model": "gpt-4o",
"provider": "openai",
"priority": 1,
"latency_budget_ms": 8000,
"max_tokens": 4096,
"supports_streaming": true,
"cost_per_1k_tokens": 0.005
},
{
"model": "claude-3-5-sonnet",
"provider": "anthropic",
"priority": 2,
"latency_budget_ms": 10000,
"max_tokens": 8192,
"supports_streaming": true,
"cost_per_1k_tokens": 0.003
},
{
"model": "llama-3.1-70b",
"provider": "together",
"priority": 3,
"latency_budget_ms": 15000,
"max_tokens": 4096,
"supports_streaming": true,
"cost_per_1k_tokens": 0.0009
}
]
}This lets you reorder, add, or remove hops without deploying code. It also makes the chain inspectable — you can log which hop served a request and why earlier hops were skipped.
Implement per-hop latency budgets
A common mistake is applying a single global timeout to the entire chain. If your first provider has a p99 of 8s and your budget is 10s, you have 2s left for the fallback — but the fallback might need 12s. Instead, assign each hop its own budget derived from its historical latency distribution:
# latency_budget.py
from dataclasses import dataclass
import time
@dataclass
class HopBudget:
hop_latency_budget_ms: int
cumulative_budget_ms: int
def compute_hop_budgets(chain: list[dict], global_budget_ms: int) -> list[HopBudget]:
"""Allocate budget proportionally to each hop's p99 latency."""
p99s = [hop["latency_budget_ms"] for hop in chain]
total_p99 = sum(p99s)
budgets = []
cumulative = 0
for p99 in p99s:
allocated = int(global_budget_ms * (p99 / total_p99))
cumulative += allocated
budgets.append(HopBudget(
hop_latency_budget_ms=allocated,
cumulative_budget_ms=min(cumulative, global_budget_ms)
))
return budgetsWhen executing the chain, pass the remaining budget to each hop, not the global budget:
async def execute_chain(chain: list[dict], request: Request, global_budget_ms: int) -> Response:
budgets = compute_hop_budgets(chain, global_budget_ms)
start = time.monotonic()
for hop, budget in zip(chain, budgets):
elapsed = int((time.monotonic() - start) * 1000)
remaining = budget.cumulative_budget_ms - elapsed
if remaining <= 0:
continue # no time left for this hop
response = await call_provider(hop, request, timeout_ms=remaining)
reason = policy.should_fallback(response, int((time.monotonic() - start) * 1000))
if reason is None:
return response.with_metadata(hop=hop["provider"], fallback_reason=None)
log_fallback(hop["provider"], reason, elapsed)
raise AllProvidersExhaustedError("fallback chain exhausted")Handle streaming responses correctly
Streaming complicates fallback because you may have already sent bytes to the client before realizing the provider is degrading. Two strategies exist:
Strategy 1: Buffer-then-forward — accumulate the full response in memory, validate, then stream to client. Simple but defeats the purpose of streaming for long outputs.
Strategy 2: Speculative streaming with cancellation — start streaming to the client immediately, but keep a reference to the upstream connection. If the provider stalls or errors mid-stream, cancel the upstream, log the partial response, and initiate fallback from the beginning with a new request.
async def stream_with_fallback(chain: list[dict], request: Request, client_stream: StreamWriter):
for hop in chain:
upstream = await connect_provider(hop, request)
try:
async for chunk in upstream.stream():
client_stream.write(chunk)
await client_stream.drain()
await upstream.close()
return # success
except ProviderError as e:
await upstream.close()
# Critical: we must restart from token 0 on the next provider
# because the client has already received partial output
request = request.with_fallback_context(previous_hop=hop["provider"], error=str(e))
continue
raise AllProvidersExhaustedError()The tradeoff: Strategy 2 means the client sees a “glitch” — the response restarts mid-sentence. For chat use cases this is often unacceptable. In that case, you need Strategy 3: provider-aware fallback where you only fall back between messages, not mid-message. This requires your client to signal “this is a new turn” so the gateway knows it’s safe to switch providers.
Health checks: active vs. passive
Passive health (inferring health from request outcomes) reacts slowly. Active health checks probe providers independently, but they consume quota and add latency. A hybrid approach works best:
# health_monitor.py
import asyncio
from dataclasses import dataclass, field
from collections import deque
import time
@dataclass
class ProviderHealth:
provider: str
consecutive_failures: int = 0
consecutive_successes: int = 0
last_success_ts: float = 0
last_failure_ts: float = 0
active_check_latency_ms: int = 0
is_healthy: bool = True
_recent_latencies: deque = field(default_factory=lambda: deque(maxlen=100))
def record_success(self, latency_ms: int):
self.consecutive_successes += 1
self.consecutive_failures = 0
self.last_success_ts = time.time()
self._recent_latencies.append(latency_ms)
if self.consecutive_successes >= 3:
self.is_healthy = True
def record_failure(self):
self.consecutive_failures += 1
self.consecutive_successes = 0
self.last_failure_ts = time.time()
if self.consecutive_failures >= 2:
self.is_healthy = False
def p99_latency_ms(self) -> int:
if not self._recent_latencies:
return 0
sorted_lat = sorted(self._recent_latencies)
return sorted_lat[int(len(sorted_lat) * 0.99)]
async def active_health_check(provider: str, interval_sec: int = 30):
while True:
start = time.monotonic()
try:
await probe_provider(provider) # minimal request, e.g. max_tokens=1
latency = int((time.monotonic() - start) * 1000)
health_registry[provider].record_success(latency)
except Exception:
health_registry[provider].record_failure()
await asyncio.sleep(interval_sec)Use the active check’s p99 latency to dynamically adjust the hop’s latency budget in the chain. If a provider’s p99 drifts from 3s to 8s, your router should automatically allocate more budget to that hop or demote it in the chain.
Preserve request identity across hops
When a request falls back, the downstream provider sees a new request. If you use request IDs for tracing, idempotency keys, or cache keys, you must propagate the original identity:
# request_context.py
from dataclasses import dataclass, field
import uuid
@dataclass
class RequestContext:
request_id: str = field(default_factory=lambda: str(uuid.uuid4()))
original_request_id: str | None = None
fallback_hops: list[str] = field(default_factory=list)
idempotency_key: str | None = None
def for_fallback(self, failed_provider: str) -> "RequestContext":
return RequestContext(
request_id=str(uuid.uuid4()),
original_request_id=self.original_request_id or self.request_id,
fallback_hops=self.fallback_hops + [failed_provider],
idempotency_key=self.idempotency_key
)This lets you answer questions like “how many requests fell back from OpenAI to Anthropic this hour?” and prevents double-charging if you retry with the same idempotency key.
Common pitfalls
1. Fallback loops
If Provider A fails, falls back to Provider B, which fails and falls back to Provider A — you’ve created a loop. Prevent this by tracking fallback_hops in the context and refusing to route to any provider already in that list.
2. Silent degradation
A provider returns 200 with garbage output (repetitive tokens, wrong language, truncated JSON). Your policy.should_fallback catches this only if you validate responses. Implement a lightweight validator per model family:
def validate_response(response: ProviderResponse, model_family: str) -> bool:
if model_family == "openai":
return response.choices and response.choices[0].finish_reason in ("stop", "length")
if model_family == "anthropic":
return response.stop_reason in ("end_turn", "max_tokens")
# Add per-family checks
return True3. Cost inversion
Your cheapest provider is at the end of the chain. Under sustained load on the primary, you silently shift 80% of traffic to the expensive fallback. Monitor fallback_rate_by_hop and alert when the primary’s share drops below a threshold.
4. Cache pollution
If you cache responses keyed only by prompt, a fallback response from Provider B pollutes the cache for Provider A. Include the provider in the cache key, or namespace caches per provider.
5. Streaming cutoff
You set a 30s timeout on the whole request. The primary streams for 25s, then stalls. The fallback has 5s to produce a complete response — impossible for long outputs. Either increase the global budget for streaming requests, or use the “fallback between messages only” strategy.
Observability you’ll need
Instrument these metrics at minimum:
| Metric | Type | Labels |
|---|---|---|
fallback_chain_duration_ms |
Histogram | chain_id, final_hop, fallback_count |
fallback_triggered_total |
Counter | chain_id, from_provider, to_provider, reason |
provider_health_status |
Gauge | provider, status (healthy/degraded/unhealthy) |
hop_latency_ms |
Histogram | provider, model, hop_position |
fallback_cost_delta_usd |
Histogram | chain_id, primary_provider, fallback_provider |
The fallback_cost_delta metric is critical — it tells you whether your fallback policy is costing you 2x or 10x the primary.
Testing the chain
Unit test the policy predicate exhaustively. Integration test the chain with a fake provider that you can program to fail in specific ways:
# test_fallback_chain.py
import pytest
from unittest.mock import AsyncMock
@pytest.mark.asyncio
async def test_fallback_on_503_then_success():
fake_openai = AsyncMock(side_effect=ProviderError(status=503))
fake_anthropic = AsyncMock(return_value=Response(text="hello"))
chain = [
{"provider": "openai", "model": "gpt-4o", "client": fake_openai},
{"provider": "anthropic", "model": "claude-3-5-sonnet", "client": fake_anthropic},
]
response = await execute_chain(chain, Request("hi"), global_budget_ms=10000)
assert response.text == "hello"
assert response.metadata["hop"] == "anthropic"
assert response.metadata["fallback_reason"] == FallbackReason.HTTP_5XX
fake_openai.assert_called_once()
fake_anthropic.assert_called_once()Load test with hey or locust against a staging chain where you inject latency and errors at the network level (tc qdisc, toxiproxy). Verify the p99 of the entire chain meets your SLO, not just the primary.
When to stop falling back
There’s a point where falling back hurts more than failing. If your chain has 5 hops and the 5th is a 70B model running on slow hardware with a 30s p99, the user experience is worse than a fast error. Define a max_chain_depth or max_cumulative_latency_ms and fail fast when exceeded. Return a structured error the client can act on:
{
"error": {
"code": "ALL_PROVIDERS_EXHAUSTED",
"message": "All providers in fallback chain failed or exceeded latency budget",
"attempted_providers": ["openai", "anthropic", "together"],
"fallback_reasons": ["http_5xx", "latency_budget_exceeded", "network_error"],
"retry_after_ms": 5000
}
}This lets the client show a meaningful message and implement its own backoff.
Summary
A production-grade provider fallback chain requires: a precise failure predicate, data-driven hop configuration, per-hop latency budgets, streaming-aware fallback semantics, hybrid health checks, request identity propagation, and observability that captures cost and latency deltas. Start with a two-hop chain (primary + one fallback), instrument heavily, and expand only when you have data showing which failure modes actually occur in your traffic. The goal isn’t to never fail — it’s to fail predictably, cheaply, and with enough context to debug.