Production LLM integrations fail in ways typical HTTP services don’t. A disciplined exponential backoff llm api retry strategy separates a flaky prototype from a system that survives provider rate limits and partial outages. This guide builds a complete retry layer in Python that you can drop into any OpenAI-compatible client.
Step 1: Classify retryable failures
Not every error should trigger a retry. A 400 from malformed JSON is permanent; a 429 from rate limiting is not. Start by enumerating the responses and exceptions you will retry.
Common retryable conditions for LLM endpoints:
- HTTP 429 (Too Many Requests)
- HTTP 500, 502, 503, 504 (upstream faults)
requests.ConnectionError,requests.Timeoutopenai.APIConnectionError,openai.RateLimitError,openai.InternalServerErrorif using the SDK
Non-retryable: 401, 403, 404, 422, and any validation error. Retrying those wastes latency and quota.
Define a predicate:
import requests
RETRYABLE_STATUS = {429, 500, 502, 503, 504}
def is_retryable(exc: Exception) -> bool:
if isinstance(exc, requests.HTTPError):
resp = exc.response
return resp is not None and resp.status_code in RETRYABLE_STATUS
if isinstance(exc, (requests.ConnectionError, requests.Timeout)):
return True
return FalseIf you use the OpenAI SDK, map its exceptions similarly. The principle stays: only retry on transient conditions.
Step 2: Write a single-shot request with explicit timeout
Never call an LLM API without a timeout. Providers can hang, especially under load. Set both connect and read timeouts.
import requests
import os
ENDPOINT = os.environ["LLM_BASE_URL"] # OpenAI-compatible
API_KEY = os.environ["LLM_API_KEY"]
def call_completion(prompt: str, model: str = "gpt-4o-mini") -> dict:
resp = requests.post(
f"{ENDPOINT}/v1/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.2,
},
timeout=(3.05, 30), # connect, read
)
resp.raise_for_status()
return resp.json()raise_for_status turns 4xx/5xx into HTTPError, which our predicate catches. The connect timeout of 3.05s avoids hanging on DNS or TCP; the read timeout of 30s caps wait on a slow generation.
Step 3: Implement the exponential backoff llm api retry loop
The core algorithm: wait base * 2**attempt seconds, add full jitter, and retry. Full jitter (random between 0 and delay) prevents thundering herds better than equal sleeps.
import time
import random
from typing import Callable, TypeVar
T = TypeVar("T")
def retry_with_backoff(
fn: Callable[[], T],
max_attempts: int = 5,
base_delay: float = 0.5,
max_delay: float = 30.0,
) -> T:
attempt = 0
while True:
try:
return fn()
except Exception as exc:
if attempt >= max_attempts - 1 or not is_retryable(exc):
raise
delay = min(base_delay * (2 ** attempt), max_delay)
sleep = random.uniform(0, delay)
time.sleep(sleep)
attempt += 1This is a minimal exponential backoff llm api retry wrapper. It retries up to max_attempts times, doubling each cycle, capped at max_delay. The first failure sleeps up to 0.5s, the second up to 1s, then 2s, 4s, 8s.
Why full jitter
If 100 clients all get a 429 and sleep exactly 1s, they retry simultaneously and trigger another 429. random.uniform(0, delay) spreads them out. AWS and Google both recommend this for client-side throttling. Equal backoff is a distributed systems anti-pattern.
Step 4: Respect Retry-After and provider hints
A 429 or 503 often includes a Retry-After header (seconds or HTTP date). Honor it instead of your computed delay when present.
Extend the predicate to extract the header:
def get_retry_after(exc: Exception) -> float | None:
if isinstance(exc, requests.HTTPError) and exc.response is not None:
val = exc.response.headers.get("Retry-After")
if val:
try:
return float(val)
except ValueError:
pass
return NoneThen in the loop:
header_delay = get_retry_after(exc)
if header_delay is not None:
sleep = min(header_delay + random.uniform(0, 1), max_delay)
else:
delay = min(base_delay * (2 ** attempt), max_delay)
sleep = random.uniform(0, delay)If you route through a gateway such as n4n.ai, which provides an OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, the gateway may already shift your request to a healthy upstream—but client retries still cover the gap before fallback completes.
Step 5: Guard against non-idempotent side effects
LLM completion calls are effectively read-only; resending the same prompt yields a new token stream but doesn’t double-charge an order. Still, if your retry wraps a function that writes to a database or triggers a webhook, you need idempotency keys.
For pure inference, pass a stable seed and temperature=0 if you need deterministic outputs across retries. The provider may still return different completions, but at least sampling is fixed.
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
"seed": 12345,
},Step 6: Wire it into a real client call
Compose the pieces:
def robust_completion(prompt: str, model: str = "gpt-4o-mini") -> dict:
return retry_with_backoff(
lambda: call_completion(prompt, model),
max_attempts=6,
base_delay=0.5,
max_delay=20.0,
)
if __name__ == "__main__":
result = robust_completion("Summarize: retries matter.")
print(result["choices"][0]["message"]["content"])For the OpenAI Python SDK, wrap client.chat.completions.create the same way. The SDK has its own retry, but it’s conservative; a custom exponential backoff llm api retry gives you control over jitter, caps, and logging.
Step 7: Add logging and metrics
Silent retries hide outages. Emit a warning on each retry with attempt number and exception type.
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("retry")
def retry_with_backoff(fn, max_attempts=5, base_delay=0.5, max_delay=30.0):
attempt = 0
while True:
try:
return fn()
except Exception as exc:
if attempt >= max_attempts - 1 or not is_retryable(exc):
raise
logger.warning("retry %d/%d due to %s", attempt+1, max_attempts, type(exc).__name__)
# ... sleep logic from Step 3/4Track counts in your metrics system (Prometheus, Datadog). A rising llm_retry_total is an early signal of provider degradation. Pair it with histogram of attempt counts to see if backoff is actually absorbing spikes.
Step 8: Test the retry path
You can’t claim reliability without a test. Use unittest.mock to simulate a 429 then success.
from unittest import mock
import requests
def test_retry_eventually_succeeds():
ok = mock.Mock()
ok.json.return_value = {"choices": [{"message": {"content": "hi"}}]}
ok.raise_for_status.return_value = None
bad = mock.Mock()
bad.response = mock.Mock(status_code=429)
bad.raise_for_status.side_effect = requests.HTTPError(response=bad.response)
with mock.patch("requests.post", side_effect=[bad, ok]):
result = robust_completion("test")
assert result["choices"][0]["message"]["content"] == "hi"Run with pytest. If the test passes, your exponential backoff llm api retry loop correctly swallowed one 429 and returned the later success. Add a second test where all attempts fail and assert the final exception propagates.
Verify success in production
Deploy the wrapper behind a low-traffic canary. Watch logs for retry warnings and confirm no MaxRetriesExceeded errors over a 24-hour window. Synthetic calls that force a 429 (via a test API key with zero quota) validate the path end to end. If you see stacked retries with no Retry-After, your jitter is working if the delay spreads.
Step 9: Consider library alternatives
Hand-rolling is educational, but tenacity is battle-tested. The same policy in tenacity:
from tenacity import retry, wait_exponential_jitter, stop_after_attempt, retry_if_exception
@retry(
wait=wait_exponential_jitter(initial=0.5, max=20.0),
stop=stop_after_attempt(6),
retry=retry_if_exception(is_retryable),
)
def tenacity_completion(prompt: str) -> dict:
return call_completion(prompt)Use it if you don’t want to maintain the loop. The concepts—retryable filter, exponential wait, jitter, cap—are identical.
Step 10: Retrying streaming responses
If you use stream=True, a failure after the first token is awkward. You can’t replay partial generation cleanly. Two patterns:
- Buffer the full response in memory and only yield after success. Defeats streaming benefit but simplifies retries.
- Wrap the stream iteration; on exception before
finish_reason, close and retry the whole request.
def stream_completion(prompt: str):
def _call():
with requests.post(f"{ENDPOINT}/v1/chat/completions",
json={"model":"gpt-4o-mini","messages":[{"role":"user","content":prompt}],"stream":True},
headers={"Authorization": f"Bearer {API_KEY}"},
timeout=(3.05, 30), stream=True) as r:
r.raise_for_status()
for line in r.iter_lines():
if line:
yield line
# retry wrapper for generators needs custom codeBecause generators can’t be restarted, you must rebuild the generator inside the retry loop. Write a function that returns a fresh generator each attempt, and only start consuming after the first chunk arrives successfully.
Final notes
An exponential backoff llm api retry layer is mandatory for any production LLM feature. Without it, a single provider hiccup becomes a user-facing outage. Implement the retryable predicate first, then the loop, then observability. Test with mocked 429s before trusting it with real traffic.
Keep timeouts strict, honor Retry-After, and log every retry. That’s the difference between hoping the API stays up and engineering for when it doesn’t.