Circuit breakers for LLM providers: keeping agents alive

A circuit breaker LLM provider pattern prevents a single degraded model endpoint from stalling your entire agent. In production agentic systems, a cascade of retries against a 429 or timeout can burn tokens and freeze tasks for minutes. This guide gives an ordered path to build, deploy, and tune that resilience layer.

Why agents break without circuit breakers

Agents loop. They call tools, reflect, call again. If the LLM provider is slow or rejecting traffic, the agent retries, often with exponential backoff that still stacks concurrent calls. Without isolation, one bad provider takes down the whole workflow.

Standard retry logic is necessary but not sufficient. Retries amplify load exactly when the downstream is struggling. A circuit breaker flips open after a failure threshold and short-circuits calls, giving the provider room to recover.

Step 1: Detect failures that matter

Not every error should trip the breaker. Classify responses before counting them:

• Transient network errors (connection reset, short timeout)
• Rate limits (429, sometimes 503 with retry-after)
• Auth/validation errors (401, 400) — these won’t fix themselves; break fast but don’t retry.
• Model-specific degradation (long latencies, partial responses)

Implement a predicate that separates breaker-worthy faults from code bugs:

import requests
from requests.exceptions import Timeout, ConnectionError

def is_breaker_error(resp: requests.Response | Exception) -> bool:
    if isinstance(resp, Exception):
        return isinstance(resp, (Timeout, ConnectionError))
    if resp.status_code == 429:
        return True
    if resp.status_code == 503 and resp.headers.get("retry-after"):
        return True
    return False

Pitfall: counting 400-level client errors as breaker trips wastes cycles. Your prompt is broken; flipping the breaker won’t help.

Step 2: Define states and thresholds

A breaker has three states: closed, open, half-open. Use these parameters:

• failure_threshold: consecutive failures to open (e.g., 5)
• reset_timeout: seconds before half-open (e.g., 30)
• success_threshold: consecutive successes in half-open to close (e.g., 2)
• latency_threshold_ms: treat slow calls as failures (e.g., 10000)

Consecutive vs sliding window

Consecutive failures are simpler and react faster to hard outages. A sliding window (e.g., 5 failures in 20 calls) tolerates intermittent blips but delays reaction. For agentic loops where one stuck call blocks a task, prefer consecutive counting with a low threshold.

Tradeoff: low thresholds trip too early on sporadic blips; high thresholds let damage accumulate. For agentic loops, err toward sensitivity—open at 5 failures.

Step 3: Implement the breaker

Minimal thread-safe Python class:

import time
import threading
from enum import Enum

class State(Enum):
    CLOSED = 0
    OPEN = 1
    HALF_OPEN = 2

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_timeout=30, success_threshold=2, latency_ms=10000):
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.success_threshold = success_threshold
        self.latency_ms = latency_ms
        self.state = State.CLOSED
        self.failures = 0
        self.successes = 0
        self.last_failure = 0
        self.lock = threading.Lock()

    def __call__(self, func):
        def wrapped(*args, **kwargs):
            with self.lock:
                if self.state == State.OPEN:
                    if time.time() - self.last_failure > self.reset_timeout:
                        self.state = State.HALF_OPEN
                        self.successes = 0
                    else:
                        raise RuntimeError("circuit open")
            start = time.time()
            try:
                resp = func(*args, **kwargs)
                if (time.time() - start) * 1000 > self.latency_ms:
                    self._record_failure()
                    raise TimeoutError("latency exceeded")
                self._record_success()
                return resp
            except Exception as e:
                self._record_failure()
                raise
        return wrapped

    def _record_failure(self):
        with self.lock:
            self.failures += 1
            self.last_failure = time.time()
            if self.state == State.HALF_OPEN:
                self.state = State.OPEN
            elif self.failures >= self.failure_threshold:
                self.state = State.OPEN

    def _record_success(self):
        with self.lock:
            if self.state == State.HALF_OPEN:
                self.successes += 1
                if self.successes >= self.success_threshold:
                    self.state = State.CLOSED
                    self.failures = 0
            else:
                self.failures = 0

If your agent runs on asyncio, port the same logic with async def and asyncio.Lock. Blocking a sync breaker inside an event loop will stall every concurrent task.

Step 4: Wire fallback routing

When the breaker is open, you must route to a secondary. Two patterns:

1. Static fallback: always call gpt-4o-mini if claude-3-5-sonnet breaker open.
2. Dynamic gateway: send a routing hint and let the gateway pick.

Local chain example:

def call_with_fallback(prompt):
    try:
        return breaker(claude_client)(prompt)
    except RuntimeError:
        # circuit open, use backup
        return gpt_client(prompt)
    except Exception:
        raise

A gateway such as n4n.ai collapses the dynamic pattern: one OpenAI-compatible endpoint fronts 240+ models and performs automatic fallback when a provider is rate-limited or degraded. You still keep a local circuit breaker LLM provider shim to avoid flooding even the gateway with doomed requests.

Pitfall: fallback models differ in prompt format, tool calling schema, and context window. Validate outputs; don’t assume parity.

Honoring cache-control

Providers support prompt caching via headers like cache-control: max-age=3600. Forward those hints on fallback calls when the secondary supports them. Losing cache on every trip doubles cost.

Step 5: Monitor and tune

Emit metrics on every state transition. A minimal JSON log line:

{
  "ts": "2025-05-12T10:22:01Z",
  "breaker": "claude-3-5-sonnet",
  "event": "open",
  "failures": 5,
  "p95_latency_ms": 11200
}

Track:

• Trip count per model
• Fallback rate (fallback calls / total calls)
• Latency distribution before trips

In agent loops, a high fallback rate signals either provider instability or too-sensitive thresholds. Tune latency_threshold_ms from observed p95 of healthy calls. If your agent tolerates 2s responses, set threshold near 3s, not 10s.

Tradeoff: aggressive timeouts improve responsiveness but increase trips during minor network jitter. Half-open probing recovers fast without a full restart.

Common pitfalls and tradeoffs

• Shared breaker across agents: a global breaker protects the provider but may starve other tenants. Scope breakers per model+tenant.
• Ignoring cache-control: fallback without cache hint spikes cost and latency.
• Synchronous blocking in async code: use an async-native breaker or run sync calls in a thread pool.
• No half-open state: permanent open requires manual reset. Always implement half-open probing.
• Blind fallback: calling a more expensive model on every trip can 10x spend. Cap fallback depth.

Putting it together

Ordered integration path for a new agent:

1. Wrap each provider call in a breaker with conservative thresholds (5 failures, 30s reset).
2. Add fallback to a cheaper or different-provider model; verify schema compatibility.
3. Meter token usage per route to spot cost leaks from repeated fallbacks.
4. Alert on breaker open > 1% of requests for a given model.
5. Review latency thresholds against real p95 every two weeks.

The circuit breaker LLM provider pattern is not optional for agentic apps that run unattended. Build it local, keep fallbacks honest, and let a gateway handle cross-provider routing so your code stays focused on the task.