A retry policy for provider outages separates a demo agent from a system that survives contact with real traffic. When an LLM endpoint returns 429 or 503, blind retries amplify load and waste tokens; a disciplined policy treats transient failures as expected noise. This guide gives an ordered path to build that policy into your agent loop without guessing.
1. Set client timeouts before anything else
A retry only helps if you know the request actually failed. Most SDKs ship with generous default timeouts—often minutes—which means a hung connection blocks your agent thread silently. Cap connect, read, and pool timeouts at values that match your latency budget.
from openai import OpenAI
import httpx
client = OpenAI(
base_url="https://api.example.com/v1", # your gateway or provider
http_client=httpx.Client(
timeout=httpx.Timeout(connect=2.0, read=30.0, write=10.0, pool=5.0)
)
)If you skip the pool timeout, you can exhaust the connection pool under retry storms and turn a provider outage into a local deadlock. Set connect low (1–2s) because DNS or TCP failures are fast. Set read based on the model’s tail latency—30s covers most chat completions but not long generations.
2. Classify errors before retrying
Not every failure is retryable. A 401 means your key is wrong; retrying it 4 times just burns latency. A 429 means you are rate-limited and need to back off. Network-level exceptions are almost always retryable.
import httpx
def is_retryable(exc: Exception) -> bool:
if isinstance(exc, httpx.TimeoutException):
return True
if isinstance(exc, httpx.ConnectError):
return True
if isinstance(exc, httpx.HTTPStatusError):
# 500/502 may be transient; 503 is provider downtime; 504 gateway timeout
return exc.response.status_code in (429, 500, 502, 503, 504)
return FalseFor 429 responses, read the Retry-After header and honor it. Ignoring it is the fastest way to get IP-banned. For 503, the service is likely overloaded; backoff is mandatory.
A common mistake is treating 400 as retryable because “maybe it was a fluke.” It wasn’t. Validate your request shape before sending, not after.
3. Exponential backoff with full jitter
Fixed-interval retries synchronize every client that fails at the same time, creating a thundering herd when the provider recovers. Exponential backoff spreads them, and full jitter removes residual correlation.
import random, time
def backoff_sleep(attempt: int, base: float = 0.5, cap: float = 8.0):
uncapped = base * (2 ** attempt)
wait = min(cap, uncapped) * random.random()
time.sleep(wait)The attempt is zero-indexed. After attempt 0, you wait up to 0.5s; after attempt 3, up to 4s (capped). Multiply by random.random() so the actual sleep is uniformly distributed between 0 and the capped value. This is simpler and more effective than equal jitter for LLM traffic patterns.
Tradeoff: full jitter increases p95 latency because some retries sleep near zero while others sleep near the cap. If you need tighter tails, use equal jitter (cap/2 + random * cap/2), but accept slightly higher herd risk.
4. Bound the retry budget
Unbounded retries are a denial-of-service against your own agent. Set a maximum attempt count and a wall-clock deadline.
def call_with_retry(fn, max_attempts=4, max_time=20.0):
deadline = time.monotonic() + max_time
attempt = 0
while True:
try:
return fn()
except Exception as e:
if not is_retryable(e) or attempt >= max_attempts or time.monotonic() > deadline:
raise
backoff_sleep(attempt)
attempt += 1Four attempts with capped backoff fits inside a 20s budget for most interactive agents. For background batch jobs, you can raise both, but never remove the deadline. A provider that is hard-down for minutes should fail fast and alert, not spin.
A robust retry policy for provider outages always includes a circuit breaker: if the last N calls failed, skip the call entirely and route elsewhere.
5. Route around dead providers
When one model is returning 503 repeatedly, retrying the same endpoint is futile. Switch to a different model or provider. If you sit behind a gateway, you can express fallback intent once.
{
"model": "gpt-4o-mini",
"route": {
"fallback_models": ["claude-3-haiku", "mistral-large"],
"on_status": [429, 503]
}
}This is illustrative, but the pattern is real: a gateway such as n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and performs automatic fallback when a provider is rate-limited or degraded. It honors client routing directives and forwards provider cache-control hints, so you can declare fallback preferences without writing branching logic in every agent. Still, keep a client-side secondary model in case the gateway itself returns a 502.
The tradeoff is cost and behavior drift. A fallback model may format JSON differently or have a smaller context. Test your prompt against every model in the fallback chain.
6. Make retries idempotent
LLM completions are not idempotent—retrying with temperature > 0 yields different text. That’s acceptable for generation, but not for side effects. If your agent calls tools (send email, write DB row), a retried completion that re-invokes the tool duplicates the action.
Use idempotency keys where the downstream API supports them:
async function postWithIdempotency(url: string, body: any, key: string) {
const res = await fetch(url, {
method: "POST",
headers: { "Content-Type": "application/json", "Idempotency-Key": key },
body: JSON.stringify(body),
});
return res;
}Generate the key per logical operation, not per HTTP attempt, so retries hit the same key. If the provider doesn’t support idempotency, make the tool itself idempotent: upsert by natural key, or check “already sent” before sending.
Also pin temperature=0 for retries on structured extraction tasks. A non-zero temperature on attempt 2 may produce a schema-valid but semantically different object, breaking downstream parsing.
7. Instrument and tune
You cannot tune a retry policy for provider outages blind. Emit per-attempt metrics: status code, sleep duration, cumulative latency, and final outcome.
import logging
logger = logging.getLogger("llm_retry")
def logged_call(fn):
attempts = 0
while True:
attempts += 1
try:
res = fn()
logger.info("llm_success", extra={"attempts": attempts})
return res
except Exception as e:
logger.warning("llm_retry", extra={"attempts": attempts, "err": str(e)})
# re-raise or sleep per policyFeed these into your dashboards. Watch the retry rate per model. If a model’s retry rate exceeds 5% sustained, it’s not “flaky”—it’s undersized for your traffic, and you should move it out of the primary slot. Watch p95 latency with retries enabled; if it doubles versus no-retry, your backoff cap is too high.
Common pitfalls
- Retrying timeouts without idempotency. A read timeout doesn’t mean the provider didn’t process the request. If the call had side effects, you’ll duplicate them.
- Hard-coding backoff without jitter. Every instance retries at 1s, 2s, 4s in lockstep. The provider recovers and instantly gets slammed again.
- Treating 400/422 as transient. These are request errors. Fix the payload; don’t loop.
- Ignoring
Retry-After. Especially on 429, the header tells you the exact wait. Sleeping less just extends the ban. - No overall deadline. A
while Truewith retryable exceptions is a hang waiting to happen.
Tradeoffs
More retries buy higher success rate at the cost of latency and token spend (you may pay for partial generations on timeout). Fallback routing improves availability but introduces output variance and possible cost spikes. Jitter reduces herd risk but loosens latency bounds.
A good retry policy for provider outages is not maximalist. It fails fast on fatal errors, backs off quietly on transient ones, switches models when a provider is down, and never hides the truth from your metrics. Build it as a small, tested module—not scattered try/except blocks around every completion call.