LLM provider endpoints fail in ways that simple retries make worse: a rate-limited gateway returns 429s, a degraded inference node hangs, and your retry storm turns a partial outage into a full one. A circuit breaker llm provider pattern cuts off calls to a misbehaving endpoint before it exhausts your worker pool, but only if you scope failures correctly and pair it with backoff. This guide walks through a concrete implementation order you can ship this week.
Why a circuit breaker belongs in your LLM integration layer
Retries are necessary because LLM APIs are eventually consistent only in the loosest sense—they drop connections, return 500s under load, and throttle aggressively. But unbounded retries against a dead dependency convert a latency spike into a thread exhaustion incident. The circuit breaker llm provider pattern adds a stateful gate: after enough failures, it fails fast locally instead of hitting the network.
This is not about replacing retries. It is about bounding their blast radius. Without a breaker, every request path waits on a timeout, holds a connection, and then retries—multiplicatively. With one, you shed load precisely when the downstream is worst.
Step 1: Define failure conditions explicitly
A breaker is only as good as its trip criteria. Count these as failures:
- HTTP 429 (rate limited)
- HTTP 5xx (server error)
- Socket timeout / deadline exceeded
- Provider-specific overload error objects (e.g.,
type: "rate_limit_error"in JSON)
Do not count 4xx validation errors (400, 401, 403) as breaker trips. A malformed request from your code should not disable the provider for everyone. If you use the OpenAI SDK, catch openai.RateLimitError, openai.APIStatusError with status >= 500, and openai.APITimeoutError.
from openai import RateLimitError, APIStatusError, APITimeoutError
def is_breaker_failure(exc: Exception) -> bool:
if isinstance(exc, RateLimitError):
return True
if isinstance(exc, APITimeoutError):
return True
if isinstance(exc, APIStatusError) and exc.status_code >= 500:
return True
return FalseStep 2: Choose state machine parameters
Three states: closed, open, half-open. Tune four numbers:
- failure_threshold: percentage or absolute count of failures in a window.
- min_calls: minimum traffic before evaluating, prevents tripping on first error.
- reset_timeout: how long to stay open before probing.
- half_open_success: consecutive successes required to close.
Typical starting point for a low-traffic service: 50% failure rate over 20 calls, open for 30s, 2 successes to close. High-traffic services should use rolling windows, not fixed counters.
Tradeoff: short reset_timeout detects recovery faster but risks repeated half-open probes against a still-broken endpoint. Long timeout protects the provider but increases user-visible degradation.
Step 3: Implement a minimal breaker
Below is a stripped-down synchronous breaker. It is not production-grade (no concurrency locks, no rolling window), but it shows the shape:
import time
class CircuitOpen(Exception):
pass
class CircuitBreaker:
def __init__(self, failure_pct=0.5, min_calls=20, reset_timeout=30):
self.failure_pct = failure_pct
self.min_calls = min_calls
self.reset_timeout = reset_timeout
self.calls = 0
self.failures = 0
self.state = "closed"
self.opened_at = 0
def __enter__(self):
if self.state == "open":
if time.time() - self.opened_at > self.reset_timeout:
self.state = "half-open"
self.calls = 0
self.failures = 0
else:
raise CircuitOpen()
return self
def __exit__(self, exc_type, exc, tb):
self.calls += 1
if exc_type is not None and is_breaker_failure(exc):
self.failures += 1
if self.state == "closed" and self.calls >= self.min_calls:
if self.failures / self.calls >= self.failure_pct:
self.state = "open"
self.opened_at = time.time()
elif self.state == "half-open":
if exc_type is not None and is_breaker_failure(exc):
self.state = "open"
self.opened_at = time.time()
else:
self.state = "closed"
return False # don't suppress exceptionsFor async or threaded services, use pybreaker or tenacity with a circuit breaker mixin. The logic stays identical.
Step 4: Wire it around the provider client
Wrap the actual completion call. Set a hard timeout on the client so hung sockets count as failures quickly:
from openai import OpenAI
client = OpenAI(timeout=10.0) # 10s deadline
breaker = CircuitBreaker(min_calls=20, reset_timeout=30)
def complete(messages):
try:
with breaker:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
)
return resp.choices[0].message.content
except CircuitOpen:
# fall through to fallback path
raiseNote the timeout=10.0 on the client constructor. Without it, a stalled TCP connection may wait minutes, defeating the breaker’s latency goal.
Step 5: Combine with retries and backoff
The breaker handles the “everyone is failing” case; retries handle the “one transient blip” case. Use a capped exponential backoff with jitter, and never retry when the breaker is open—the CircuitOpen exception should propagate immediately.
import random, time
def complete_with_retry(messages, max_attempts=3):
for attempt in range(max_attempts):
try:
return complete(messages)
except CircuitOpen:
raise # breaker says stop
except (RateLimitError, APITimeoutError, APIStatusError) as e:
if attempt == max_attempts - 1:
raise
sleep = min(2 ** attempt + random.uniform(0, 0.5), 8)
time.sleep(sleep)Pitfall: placing retries inside the __enter__/__exit__ context causes the breaker to count each retry as a separate call, tripping prematurely. Keep retries outside the breaker scope.
Step 6: Handle fallback and degradation
When the breaker is open, your service must do something useful:
- Return a cached prior response for identical prompts (hash the message list).
- Route to a smaller local model.
- Queue the request for later processing if the task is asynchronous.
If you front your calls with a gateway like n4n.ai, its automatic fallback when a provider is rate-limited or degraded can mask some outages, but your own circuit breaker llm provider layer remains essential to avoid saturating your client workers with blocked sockets. The gateway operates at the network edge; you operate at the process boundary.
Step 7: Observe and tune
Emit three metrics:
breaker_state(0=closed, 1=half-open, 2=open)breaker_trip_count(counter)llm_request_latencytagged by state
A breaker that trips constantly indicates either a real provider problem or mis-scoped failure detection (e.g., counting 400s). A breaker that never trips suggests min_calls too high or failure_pct too lenient. Alert on trip rate exceeding 1 per minute per provider.
Common pitfalls
Counting client errors as failures. A bad API key returns 401. Tripping the breaker on that wastes 30 seconds of avoidable downtime.
Ignoring latency. A provider returning 200 after 25 seconds is often worse than a 500. Add a latency threshold: if elapsed > soft_limit, record a failure even on success.
Sharing one breaker across models. gpt-4o and gpt-4o-mini may have independent quotas. Use a breaker keyed by model + provider.
No half-open success criterion. Simply flipping to closed after timeout causes a full traffic blast that re-trips instantly. Require N consecutive successes.
Tradeoffs summary
Adding a circuit breaker llm provider layer introduces a small amount of stateful complexity and a failure mode where healthy traffic is blocked during the open window. That cost is negligible compared to cascading worker exhaustion. The pattern is mature—borrowed from resilience engineering for databases—and maps cleanly onto inference endpoints because they exhibit the same throttle-and-hang behavior.
Ship the breaker with conservative defaults, expose its state in your health endpoint, and adjust thresholds from real error histograms rather than guesswork.