Circuit breakers llm api reliability are client-side or gateway-side state machines that trip after a threshold of failures or slow responses from a model endpoint, then block calls for a cool-down window. This pattern adapts the classic distributed systems circuit breaker to LLM inference, where each failed call can burn tokens and inflate latency.
What circuit breakers llm api reliability mean
A circuit breaker watches calls to a dependency and flips between three states based on observed health. In the LLM context the dependency is a model endpoint—often reached through an OpenAI-compatible gateway—rather than a conventional microservice.
The three states
- Closed: Calls flow normally. The breaker counts failures within a rolling window.
- Open: The failure threshold tripped. Calls are rejected immediately without hitting the network.
- Half-open: After a reset timeout, the breaker allows a single probe call (or a small number) to test recovery. Success closes the breaker; failure re-opens it.
This is identical in structure to the pattern described by Nygard in Release It!, but the failure semantics differ.
Failure signals specific to LLMs
HTTP 500/502/503 from the provider are obvious. So is a socket timeout. But LLM APIs introduce softer failures:
- HTTP 429 (rate limit) with
Retry-Afterheaders. - Truncated responses where
finish_reason == "length"but your contract expected full JSON. - Content filter blocks returning 200 with an empty completion.
- Schema validation errors when you asked for structured output and got prose.
A robust breaker counts these as failures, or at least as degraded signals that feed a separate latency/quality budget.
Why they matter more than for standard REST
Traditional REST calls are cheap and fast. An LLM completion can cost cents and take 10–30 seconds for a large prompt. Repeatedly hammering a degraded endpoint is not just slow—it drains your token budget and can get your API key throttled harder.
Token burn
If a provider is returning 500 after 20 seconds, each retry spends input tokens and wastes compute. A breaker that opens after three failures saves the fourth, fifth, and sixth attempts entirely.
Latency amplification
Your user is waiting on a synchronous chain. If your service retries five times with exponential backoff against a dead model, you’ve added minutes of latency. The breaker short-circuits to a fallback path in milliseconds.
Cascading degradation
In a gateway setup, one slow provider can saturate your worker pool. Gateways such as n4n.ai provide automatic fallback across providers when one is degraded, yet a local breaker remains essential to stop retrying a known-bad route and exhausting connection limits.
How to implement one
You do not need a heavy framework. A minimal Python class handles the core logic.
import time
class CircuitBreaker:
def __init__(self, failure_threshold=5, reset_timeout=30, window=10):
self.failures = 0
self.failure_threshold = failure_threshold
self.reset_timeout = reset_timeout
self.window = window # seconds for rolling count
self.state = "closed"
self.opened_at = 0
self.last_failure = 0
def call(self, func, *args, **kwargs):
if self.state == "open":
if time.time() - self.opened_at > self.reset_timeout:
self.state = "half-open"
else:
raise RuntimeError("circuit open")
try:
result = func(*args, **kwargs)
if self.state == "half-open":
self.state = "closed"
self.failures = 0
return result
except Exception:
self.failures += 1
self.last_failure = time.time()
if self.failures >= self.failure_threshold:
self.state = "open"
self.opened_at = time.time()
raiseThis is intentionally naive: it uses a fixed counter, not a sliding window. For production, track timestamps and evict old failures.
Sliding window vs fixed counter
A fixed counter resets only when the breaker opens. Under bursty traffic, a single bad minute can trip it, then a lull resets the count. A rolling window (e.g., last 20 seconds) reflects current health. Use a collections.deque of failure timestamps:
from collections import deque
class SlidingBreaker:
def __init__(self, threshold=5, window=20, reset_timeout=30):
self.fails = deque()
self.threshold = threshold
self.window = window
self.reset_timeout = reset_timeout
self.state = "closed"
self.opened_at = 0
def _clean(self):
now = time.time()
while self.fails and now - self.fails[0] > self.window:
self.fails.popleft()
def allow(self):
if self.state == "open":
if time.time() - self.opened_at > self.reset_timeout:
self.state = "half-open"
return True # permit one probe
return False
return True
def record_failure(self):
self._clean()
self.fails.append(time.time())
if len(self.fails) >= self.threshold:
self.state = "open"
self.opened_at = time.time()
def record_success(self):
if self.state == "half-open":
self.state = "closed"
self.fails.clear()Concrete integration example
Wire the breaker around an OpenAI-compatible client. The example points at a unified endpoint that fronts many models.
import openai
client = openai.OpenAI(
base_url="https://api.n4n.ai/v1", # OpenAI-compatible, 240+ models
api_key="sk-your-key"
)
breaker = SlidingBreaker(threshold=3, window=15, reset_timeout=20)
def complete(prompt: str):
if not breaker.allow():
raise RuntimeError("breaker open; skip call")
try:
resp = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": prompt}]
)
breaker.record_success()
return resp.choices[0].message.content
except Exception:
breaker.record_failure()
raiseWhen the provider behind that route returns sustained 429s or 5xx, the breaker opens after three failures in 15 seconds and stops sending requests for 20 seconds. Your application can catch RuntimeError and fall back to a smaller local model or a cached answer.
Common misconceptions
Breakers replace retries
They do not. Retries with exponential backoff handle transient blips. A circuit breaker handles sustained outages. Use both: retry a couple times with jitter, then let the breaker trip if the dependency stays unhealthy.
Any 429 should trip the breaker
Rate limits are expected at scale. A single 429 with a short Retry-After is not a breaker event. Trip only on sustained 429s (e.g., 10 in a row) or when the Retry-After exceeds your SLA budget.
One global breaker suffices
You need per-model, per-provider isolation. Tripping on gpt-4o should not block mistral-small. If you route through a gateway that honors client routing directives, keep a breaker per logical route key.
Half-open is safe
A half-open probe still hits the dependency. If your probe is a huge 8k-token completion, you just spent money to check health. Make probes cheap: a tiny heartbeat prompt or a lightweight embedding call.
Tuning for production
Set thresholds from real traffic. If you call a model 100 times per minute, a threshold of 5 failures in 10 seconds is reasonable. If you call it twice per minute, that same threshold is too strict.
Reset timeouts should match provider recovery times. LLM providers often recover in seconds, but regional degradations can last minutes. Start with 20–30 seconds and adjust from incident reviews.
Finally, emit metrics. Log state transitions and failure counts to Prometheus or whatever you use. Without visibility, a breaker is a silent black hole that masks outages instead of surfacing them.
Circuit breakers llm api reliability are not magic, but they are mandatory once you run LLMs in a latency-sensitive or cost-bound production path. Implement them per route, pair them with smart retries, and watch the metrics.