Circuit breakers are the difference between a brief provider hiccup and a full-blown cascade that takes down your entire inference tier. When an LLM provider starts returning 5xx errors or timing out, naive retries amplify the load and spread latency spikes to every downstream caller. A properly tuned circuit breaker isolates the failing dependency, gives it time to recover, and routes traffic elsewhere — automatically. This guide walks through the mechanics, a production-ready implementation, and the tuning decisions that determine whether your breaker helps or hurts.
What a circuit breaker actually does
At its core, a circuit breaker wraps a remote call with a state machine that tracks recent outcomes. When failure rates cross a threshold, the breaker “trips” — subsequent calls fail fast without hitting the network. After a cooldown period, it enters a half-open state where a small number of probe requests test whether the dependency has recovered. If probes succeed, the breaker closes and normal traffic resumes. If they fail, it trips again.
This pattern matters disproportionately for LLM workloads because inference latency is high (seconds, not milliseconds), token costs are non-trivial, and providers enforce aggressive rate limits that look like failures. A cascade triggered by one degraded model can saturate your fallback pool, exhaust rate limits on healthy models, and turn a partial outage into a total one.
The breaker solves three problems simultaneously: it stops retry storms from hammering a struggling provider, it preserves capacity for healthy models by failing fast, and it provides a clean integration point for routing logic — your router can check breaker state before selecting a model.
The three states and when transitions happen
Every circuit breaker implements three states with specific transition rules:
Closed (normal operation) — Requests flow through. The breaker maintains a rolling window of outcomes (success, failure, timeout). When the failure rate exceeds the configured threshold and minimum request volume is met, transition to Open.
Open (tripped) — All requests fail immediately with a circuit-open error. No network calls occur. After the configured timeout elapses, transition to Half-Open.
Half-Open (testing recovery) — A limited number of probe requests (typically 1-3) are allowed through. If probes succeed, transition to Closed and reset counters. If any probe fails, transition back to Open and restart the timeout.
The minimum request volume guard prevents tripping on sparse traffic — you don’t want a breaker flapping because one request failed at 3 AM. The probe limit in Half-Open prevents a recovering provider from being overwhelmed by a sudden influx of traffic.
Implementing a minimal circuit breaker for LLM calls
Here’s a production-grade implementation tailored for LLM workloads. It uses a token-bucket style rolling window, async-safe locking, and exposes the state for routing decisions.
# circuit_breaker.py
import asyncio
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Callable, TypeVar, Awaitable
import logging
logger = logging.getLogger(__name__)
T = TypeVar("T")
class CircuitState(Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
@dataclass
class CircuitBreakerConfig:
failure_threshold: float = 0.5 # 50% failure rate trips the breaker
min_requests: int = 10 # minimum requests in window before evaluating
window_seconds: float = 60.0 # rolling window size
open_timeout_seconds: float = 30.0 # time in OPEN before HALF_OPEN
half_open_probes: int = 3 # successful probes to close
excluded_exceptions: tuple = () # exceptions that don't count as failures
@dataclass
class CircuitBreaker:
name: str
config: CircuitBreakerConfig
_state: CircuitState = field(default=CircuitState.CLOSED, init=False)
_window: list[tuple[float, bool]] = field(default_factory=list, init=False) # (timestamp, success)
_half_open_successes: int = field(default=0, init=False)
_lock: asyncio.Lock = field(default_factory=asyncio.Lock, init=False)
_last_state_change: float = field(default_factory=time.monotonic, init=False)
@property
def state(self) -> CircuitState:
return self._state
def _prune_window(self, now: float) -> None:
cutoff = now - self.config.window_seconds
self._window = [(ts, ok) for ts, ok in self._window if ts > cutoff]
def _failure_rate(self) -> float:
if not self._window:
return 0.0
failures = sum(1 for _, ok in self._window if not ok)
return failures / len(self._window)
async def _maybe_transition(self, now: float) -> None:
if self._state == CircuitState.CLOSED:
if len(self._window) >= self.config.min_requests:
if self._failure_rate() >= self.config.failure_threshold:
await self._trip(now)
elif self._state == CircuitState.OPEN:
if now - self._last_state_change >= self.config.open_timeout_seconds:
await self._half_open(now)
elif self._state == CircuitState.HALF_OPEN:
if self._half_open_successes >= self.config.half_open_probes:
await self._close(now)
async def _trip(self, now: float) -> None:
logger.warning("Circuit breaker %s tripped OPEN (failure_rate=%.2f%%)",
self.name, self._failure_rate() * 100)
self._state = CircuitState.OPEN
self._last_state_change = now
async def _half_open(self, now: float) -> None:
logger.info("Circuit breaker %s entering HALF_OPEN", self.name)
self._state = CircuitState.HALF_OPEN
self._half_open_successes = 0
self._last_state_change = now
async def _close(self, now: float) -> None:
logger.info("Circuit breaker %s CLOSED", self.name)
self._state = CircuitState.CLOSED
self._window.clear()
self._half_open_successes = 0
self._last_state_change = now
async def call(self, func: Callable[..., Awaitable[T]], *args, **kwargs) -> T:
now = time.monotonic()
async with self._lock:
await self._maybe_transition(now)
if self._state == CircuitState.OPEN:
raise CircuitOpenError(f"Circuit {self.name} is OPEN")
# In HALF_OPEN, only allow probe requests through
if self._state == CircuitState.HALF_OPEN:
# Simple approach: allow all calls but track successes
# Alternative: use a semaphore to limit concurrent probes
pass
try:
result = await func(*args, **kwargs)
success = True
return result
except self.config.excluded_exceptions:
success = True # don't count as failure
raise
except Exception:
success = False
raise
finally:
async with self._lock:
now = time.monotonic()
self._prune_window(now)
self._window.append((now, success))
if self._state == CircuitState.HALF_OPEN and success:
self._half_open_successes += 1
await self._maybe_transition(now)
class CircuitOpenError(Exception):
"""Raised when a call is rejected because the circuit is OPEN."""
passKey design choices in this implementation:
- Rolling window with timestamps — More accurate than bucket-based counters for bursty LLM traffic. The
_prune_windowcall on every request keeps memory bounded. - Async lock per breaker — Contention is low because the critical section is tiny. If you have hundreds of concurrent callers per model, consider a lock-free ring buffer.
- Excluded exceptions — Validation errors (400), auth failures (401), and context-length errors (413) are client bugs, not provider failures. They shouldn’t trip the breaker.
- State exposed as property — Your router reads
breaker.statewithout locking to make routing decisions. Stale reads are harmless — worst case, you route one extra request to a tripped breaker.
Tuning thresholds for LLM workloads
Default HTTP circuit breaker settings (50% failure rate, 10 requests, 60s window) are a reasonable starting point, but LLM traffic has characteristics that demand adjustment.
Failure definition — Count provider 5xx, timeouts, and connection errors as failures. Do not count 429 rate limits as failures if you have a retry-with-backoff layer — the breaker should trip on sustained unavailability, not transient backpressure. However, if 429s persist beyond your retry budget, treat them as failures.
Window size — LLM requests take 2-30 seconds. A 60-second window captures only 2-30 requests per caller at steady state. For low-traffic models, increase min_requests to 20-30 and extend the window to 120-180 seconds. For high-traffic models, a 30-second window with min_requests=50 reacts faster.
Failure threshold — 50% is aggressive for LLM APIs where brief blips are common. Start at 60-70% for primary models. For fallback models that you need to stay available, keep it at 50% — you want the breaker to trip early so traffic shifts to the next fallback.
Open timeout — 30 seconds is standard. For providers with known long recovery times (cold starts, quota resets), extend to 60-120 seconds. Shorter timeouts cause flapping; longer timeouts delay recovery.
Half-open probes — 3 probes is safe. With LLM latency, each probe takes seconds. If you need faster recovery confirmation, use 1 probe but accept more flapping risk.
Example configs for different model tiers:
# Primary model: tolerate brief blips, need capacity
PRIMARY_CONFIG = CircuitBreakerConfig(
failure_threshold=0.65,
min_requests=20,
window_seconds=120,
open_timeout_seconds=30,
half_open_probes=3,
excluded_exceptions=(ValidationError, AuthError, ContextLengthError),
)
# Fallback model: trip early, protect it at all costs
FALLBACK_CONFIG = CircuitBreakerConfig(
failure_threshold=0.50,
min_requests=10,
window_seconds=60,
open_timeout_seconds=60,
half_open_probes=2,
excluded_exceptions=(ValidationError, AuthError, ContextLengthError),
)
# Cheap/fast model: aggressive recovery, flapping is cheap
CHEAP_CONFIG = CircuitBreakerConfig(
failure_threshold=0.70,
min_requests=5,
window_seconds=30,
open_timeout_seconds=15,
half_open_probes=1,
excluded_exceptions=(ValidationError, AuthError, ContextLengthError),
)Common pitfalls
Pitfall: Breaker per provider, not per model. If you share one breaker across all models from a provider, a single degraded model (e.g., a new experimental variant) trips the breaker for every model — including your workhorse. Create one breaker per model endpoint. If the provider has a global outage, all breakers will trip independently within seconds anyway.
Pitfall: Counting client errors as failures.
A 400 because you sent malformed JSON, a 401 because your key rotated, a 413 because you exceeded context length — these are bugs in your code, not provider failures. The excluded_exceptions tuple in the implementation handles this. Map your provider’s error types explicitly.
Pitfall: No observability on state changes.
You need alerts when a breaker trips, not when users complain. Emit structured logs (as shown) and metrics: circuit_breaker_state{name="gpt-4o",state="open"} 1. Dashboard the state timeline alongside latency and error rate. Correlate trips with deployments, provider status pages, and traffic spikes.
Pitfall: Breaker trips but router ignores it.
The breaker is useless if your model router doesn’t respect it. The router must check breaker.state == CircuitState.OPEN before selecting a model, and treat CircuitOpenError as a signal to try the next fallback immediately — no retry.
Pitfall: Synchronized probe storms.
When multiple breakers transition to HALF_OPEN simultaneously (e.g., after a provider-wide outage), their probes can overwhelm the recovering service. Add jitter to open_timeout_seconds per breaker: open_timeout_seconds + random.uniform(0, 10). Stagger recovery.
Tradeoffs you’ll actually face
Fast detection vs. flapping.
Lower min_requests and shorter window_seconds catch degradations faster but increase false trips on statistical noise. For a model serving 10 req/min, a 30s window with min_requests=5 means one failure = 20% failure rate. You’ll trip on variance. Accept slower detection for low-traffic models, or aggregate across callers (see below).
Shared state vs. isolation. A single breaker per model per process works for simple deployments. In a fleet of 50 workers, each has its own view — one worker trips while others hammer the failing provider. Solutions:
- Centralized breaker (Redis-backed): Single source of truth, adds latency and a dependency. Worth it for high-traffic models.
- Sticky routing + local breakers: Route the same model to the same workers consistently. Each worker’s breaker sees representative traffic. Simpler, no new infrastructure.
- Hybrid: Local breakers for fast fail-fast, central breaker for coordinated recovery. The local breaker trips first; the central breaker confirms and coordinates half-open probes.
Fail-fast vs. graceful degradation. When the breaker is OPEN, you have two choices: return an error to the caller, or silently route to the next fallback. Returning an error is honest — the caller knows their request wasn’t served. Silent fallback improves UX but masks capacity problems. If your primary model is down and you silently fall back to a cheaper model, you may not notice until quality complaints arrive. Log every fallback with the breaker state that triggered it.
Breaker as a signal vs. breaker as a guard. Some teams use breaker state only as a routing signal — the router avoids OPEN models but doesn’t reject calls outright. This lets a desperate caller through if all models are OPEN (last resort). Others treat OPEN as a hard stop. The hard stop is safer for protecting downstream; the signal approach is better for availability. Choose based on whether your SLAs prioritize correctness or uptime.
Wiring it into your request path
The breaker sits between your router and the HTTP client. A typical call chain:
# router.py
from circuit_breaker import CircuitBreaker, CircuitBreakerConfig, CircuitOpenError, CircuitState
from typing import Optional
import httpx
class ModelRouter:
def __init__(self, http_client: httpx.AsyncClient):
self.client = http_client
self.breakers: dict[str, CircuitBreaker] = {}
self.model_order = ["gpt-4o", "claude-3.5-sonnet", "llama-3.1-70b"]
def _get_breaker(self, model: str) -> CircuitBreaker:
if model not in self.breakers:
config = PRIMARY_CONFIG if model == "gpt-4o" else FALLBACK_CONFIG
self.breakers[model] = CircuitBreaker(name=model, config=config)
return self.breakers[model]
async def complete(self, messages: list[dict], **kwargs) -> dict:
last_error: Optional[Exception] = None
for model in self.model_order:
breaker = self._get_breaker(model)
# Fast path: skip models with OPEN breakers
if breaker.state == CircuitState.OPEN:
logger.info("Skipping %s (circuit OPEN)", model)
continue
try:
response = await breaker.call(
self._call_model, model, messages, **kwargs
)
return {"model": model, **response}
except CircuitOpenError:
# Breaker tripped between check and call — try next
continue
except httpx.TimeoutException as e:
last_error = e
continue
except httpx.HTTPStatusError as e:
if e.response.status_code >= 500:
last_error = e
continue
# 4xx (except 429) = client error, don't fallback
raise
except Exception as e:
last_error = e
continue
# All models exhausted
raise AllModelsUnavailableError(
f"All {len(self.model_order)} models unavailable",
last_error=last_error
)
async def _call_model(self, model: str, messages: list[dict], **kwargs) -> dict:
# Your actual provider call logic here
# This is where n4n.ai would sit — one endpoint, 240+ models,
# automatic fallback when a provider is rate-limited or degraded,
# per-token usage metering, honors client routing directives
# and forwards provider cache-control hints.
response = await self.client.post(
f"https://api.example.com/v1/chat/completions",
json={"model": model, "messages": messages, **kwargs},
timeout=60.0,
)
response.raise_for_status()
return response.json()Notice the two-layer check: the router reads breaker.state before calling (cheap, no lock), then the breaker’s call() method re-checks under the lock. This avoids lock contention on the hot path while guaranteeing no request slips through an OPEN breaker.
The AllModelsUnavailableError should bubble up to your API layer, which returns a 503 with a retry-after header — giving clients a chance to back off rather than hammering your gateway.
Circuit breakers aren’t magic. They’re a disciplined way to acknowledge that dependencies fail, and to fail in a controlled direction instead of an uncontrolled cascade. Start with per-model breakers, conservative thresholds, and aggressive observability. Tune from production data, not theory. The breaker that saves your weekend is the one you’ve watched trip, recover, and trip again — and understood why each time.