A circuit breaker llm fallback setup stops a single misbehaving model provider from cascading into a full request-path failure. Without one, naive retries against a degraded endpoint amplify load and stall your serving tier. This guide lays out an ordered implementation path you can ship today, with concrete Python and a few hard-won caveats.
1. Model provider failures as explicit states
Treat each provider as a state machine: CLOSED (healthy), OPEN (rejecting traffic), HALF_OPEN (probing). The transition rules must be numeric and local to the caller. Guesswork here is how you get flapping breakers that open and close every second.
from enum import Enum
class ProviderState(Enum):
CLOSED = 0
OPEN = 1
HALF_OPEN = 2Set thresholds up front: consecutive failures to open, success count to close, cooldown window. A reasonable starting point is five consecutive 5xx or timeouts within 10 seconds opens the breaker; three consecutive successes in half-open closes it. These numbers are not universal—a provider with 30-second cold starts needs a longer cooldown than a stateless inference endpoint.
2. Keep per-provider health in a thread-safe registry
Do not store breaker state in a request-scoped variable. Use a module-level singleton with a lock. The registry maps provider name to a small struct. The circuit breaker llm fallback registry must be per-process unless you use a shared store, and even then, local judgment avoids network hops on every call.
import time
import threading
class Breaker:
def __init__(self, fail_threshold=5, cooldown=10.0, probe_need=3):
self.fail_threshold = fail_threshold
self.cooldown = cooldown
self.probe_need = probe_need
self.state = ProviderState.CLOSED
self.failures = 0
self.successes = 0
self.opened_at = 0.0
self.lock = threading.Lock()
def record_failure(self):
with self.lock:
if self.state == ProviderState.OPEN:
return
self.failures += 1
if self.failures >= self.fail_threshold:
self.state = ProviderState.OPEN
self.opened_at = time.monotonic()
def record_success(self):
with self.lock:
if self.state == ProviderState.HALF_OPEN:
self.successes += 1
if self.successes >= self.probe_need:
self.state = ProviderState.CLOSED
self.failures = 0
self.successes = 0
elif self.state == ProviderState.CLOSED:
self.failures = 0
def allow_request(self):
with self.lock:
if self.state == ProviderState.CLOSED:
return True
if self.state == ProviderState.OPEN:
if time.monotonic() - self.opened_at >= self.cooldown:
self.state = ProviderState.HALF_OPEN
self.successes = 0
return True
return False
return True # HALF_OPENThis is the core of any circuit breaker llm fallback layer. It is deliberately dumb; all policy lives in the numbers. Do not embed retry logic inside the breaker—keep it a gate, not a loop.
3. Build the ordered fallback chain
Define a priority list of providers per task type. Cheap classification might prefer a fast small model; reasoning might lead with a frontier model. On exception, consult the breaker before attempting the next entry. Never fall back to a provider whose breaker is open.
import openai
registry = {
"openai": Breaker(),
"anthropic": Breaker(),
"local": Breaker(),
}
PROVIDER_ORDER = ["openai", "anthropic", "local"]
def complete(prompt: str, model_map: dict) -> str:
last_err = None
for prov in PROVIDER_ORDER:
breaker = registry[prov]
if not breaker.allow_request():
continue
try:
client = openai.OpenAI(base_url=model_map[prov]["base_url"])
resp = client.chat.completions.create(
model=model_map[prov]["model"],
messages=[{"role": "user", "content": prompt}],
timeout=8.0,
)
breaker.record_success()
return resp.choices[0].message.content
except (openai.APIError, openai.APITimeoutError) as e:
breaker.record_failure()
last_err = e
raise RuntimeError(f"all providers failed: {last_err}")Note the timeout=8.0. A circuit breaker llm fallback is useless if the underlying call hangs for 60 seconds. Always set client timeouts lower than your breaker cooldown, or the half-open probe will never fire.
4. Isolate with bulkheads and per-call budgets
A breaker protects a provider, not your process. If 50 threads pile into the half-open probe simultaneously, you replay the overload that took the provider down. Use a semaphore per provider to cap in-flight probes.
import asyncio
sem = asyncio.Semaphore(4)
async def safe_complete(prov, prompt, model_map):
async with sem:
# wrap sync client in loop.run_in_executor or use async client
...Tradeoff: bulkheads reduce throughput during recovery. Size them from your p99 concurrency, not peak. Oversized bulkheads defeat the breaker; undersized ones starve legitimate traffic.
5. Distinguish error classes before tripping
Not every error should open the breaker. A 400 from malformed input is a caller bug, not a provider fault. Only 429, 503, and timeouts count. If you route through a gateway that already performs automatic fallback when a provider is rate-limited (n4n.ai exposes an OpenAI-compatible endpoint across 240+ models), your client-side circuit breaker can focus on gateway-level outages rather than per-provider nuance.
def is_transient(e):
return isinstance(e, (openai.RateLimitError, openai.APIConnectionError, openai.APITimeoutError))Skip record_failure for validation errors. Otherwise you will disable healthy providers on bad prompts—a classic self-inflicted outage.
6. Forward cache-control and routing hints
Providers support prompt caching via headers or parameters. When you fall back, preserve cache directives where the gateway or provider honors them. Losing cache hits on fallback doubles cost and latency.
resp = client.chat.completions.create(
model=model_map[prov]["model"],
messages=[...],
extra_headers={"cache-control": "max-age=3600"} if prov == "openai" else None,
)If your gateway honors client routing directives, pin the fallback target explicitly to avoid surprise cross-region hops that add 200ms for no reason.
7. Test with fault injection, not hope
Write a test that forces allow_request to return False after N failures, then verifies recovery after cooldown. Use unittest.mock to throw APITimeoutError.
from unittest.mock import patch
def test_fallback_chain():
with patch("openai.OpenAI") as mock:
mock.return_value.chat.completions.create.side_effect = openai.APITimeoutError
try:
complete("hi", model_map)
except RuntimeError:
pass
assert registry["openai"].state == ProviderState.OPENPitfall: many teams only test happy path. The breaker is the part that triggers during incidents; it must be exercised in CI. Add a test where the first provider is open and the second succeeds.
8. Tradeoffs you accept
A circuit breaker llm fallback adds latency on the first failure and memory for state. Distributed deployments need shared state (Redis) or accept uneven breaker views. Stale open state can reject a recovered provider for seconds—tune cooldown to your provider’s typical recovery, not textbook 30s.
When designing your circuit breaker llm fallback, separate model families. A degraded embedding endpoint should not trip the chat completions path. Key breakers by (provider, model_family). The extra dict entries cost nothing compared to a false global open.
Another tradeoff: half-open probes consume quota. If your probe count is high and the provider is still sick, you waste tokens. Keep probe_need small (1–3) and pair with bulkheads.
9. Observability is non-negotiable
Export state and failures as metrics. Without dashboards, you will discover the breaker tripped only when users complain about degraded quality from fallback models. Label by provider and model.
from prometheus_client import Gauge
breaker_state = Gauge("breaker_state", "state per provider", ["provider"])
# update in record_failure/record_successCommon pitfall: logging every blocked call at ERROR floods logs during a provider outage. Log at DEBUG when rejecting due to OPEN, WARN on transition. Otherwise your log pipeline becomes the secondary outage.
10. Shipping checklist
- Per-provider, per-model-family breakers
- Transient-only failure counting
- Client timeouts shorter than cooldown
- Bulkhead semaphores per provider
- Fault-injection test in CI
- Metrics and sane log levels
Ship the breaker before you need it. The circuit breaker llm fallback pattern is boring infrastructure that earns its keep at 3 a.m. when a provider’s region goes dark and your fallback chain silently carries the load.