Building exponential backoff for LLM API retries

When you build against LLM APIs at scale, transient 429s and 5xxs are inevitable. A naive retry loop will amplify load and get you throttled harder; proper exponential backoff llm api retries dampens spikes and keeps p99 latencies sane. This post walks through building a production-grade retry layer from scratch, with runnable Python and concrete failure modes you will hit in production.

Step 1: Classify errors before retrying

Retry only what is safe. HTTP 429 (rate limit), 500, 502, 503, 504, and network-level errors (connection reset, timeout) are retryable. Authentication failures (401), bad requests (400), and not-found (404) are not—retrying wastes quota and masks bugs in your prompt schema.

import requests
from requests.exceptions import ConnectionError, Timeout

def is_retryable(status_code, exc=None):
    if exc is not None and isinstance(exc, (ConnectionError, Timeout)):
        return True
    return status_code in {429, 500, 502, 503, 504}

If your LLM call includes side-effecting tool use (writing to a database, calling a webhook), ensure the operation is idempotent before looping. Inference alone is read-only, but agentic workflows are not. A retry that triggers a second bank transfer is a incident, not a resilience win.

Step 2: Implement the core backoff math

The backbone of exponential backoff llm api retries is a delay that grows geometrically with attempt count. Start with a base of 500 ms and a factor of 2. Cap the maximum delay at something human: 30s is plenty for LLM APIs because generation itself can take 20–40s.

def backoff_delay(attempt, base=0.5, factor=2.0, max_delay=30.0):
    # attempt is 0-indexed
    return min(base * (factor ** attempt), max_delay)

A minimal loop without jitter looks like this:

import time

def call_with_retries(fn, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            resp = fn()
            if resp.status_code == 200:
                return resp
            if not is_retryable(resp.status_code):
                resp.raise_for_status()
        except (ConnectionError, Timeout):
            if attempt == max_attempts - 1:
                raise
        delay = backoff_delay(attempt)
        time.sleep(delay)
    raise RuntimeError("exhausted retries")

This works for a single client, but synchronized sleep across thousands of containers triggers a thundering herd when the service recovers. The next step fixes that.

Step 3: Add jitter to avoid thundering herds

Always randomize the delay. Full jitter (random() * raw) is simplest and effective. Equal jitter (raw/2 + random()*raw/2) keeps some predictability for debugging. Decorrelated jitter (multiply previous delay by random between 1 and 3) is better when you expect long outages.

import random

def jittered_delay(attempt, base=0.5, factor=2.0, max_delay=30.0):
    raw = backoff_delay(attempt, base, factor, max_delay)
    return raw / 2 + random.uniform(0, raw / 2)

Replace time.sleep(delay) with time.sleep(jittered_delay(attempt)). For most LLM gateways, equal jitter is fine because requests are independent and latency tolerance is seconds, not microseconds. The key point: never let two failed clients wake up at the exact same millisecond.

Step 4: Honor Retry-After and provider hints

Providers often send a Retry-After header (in seconds or HTTP date) on 429/503. Use the max of your computed delay and that value. If you sit behind a gateway such as n4n.ai, it forwards provider cache-control and rate-limit hints, so parse those headers before falling back to local math. Ignoring them is the fastest way to get your IP permanently capped.

from datetime import datetime
from email.utils import parsedate_to_datetime

def parse_retry_after(headers):
    if 'Retry-After' in headers:
        val = headers['Retry-After']
        try:
            return float(val)
        except ValueError:
            return (parsedate_to_datetime(val) - datetime.now()).total_seconds()
    return 0

# inside loop:
delay = max(jittered_delay(attempt), parse_retry_after(resp.headers))

Some providers also send X-RateLimit-Reset; treat it the same way. Your backoff should be a floor, not a ceiling, when the server tells you exactly when to come back.

Step 5: Bound retries with deadlines, not just counts

Count-based retries lie about user experience. A request that retries five times at 8s each blows a 10s user timeout. Track elapsed wall-clock and abort if exceeding a budget derived from your upstream SLA.

from time import monotonic

def call_with_budget(fn, max_attempts=5, total_budget=30.0):
    start = monotonic()
    for attempt in range(max_attempts):
        if monotonic() - start > total_budget:
            raise TimeoutError("retry budget exhausted")
        # ... try/except as before
        delay = jittered_delay(attempt)
        if monotonic() + delay - start > total_budget:
            break
        time.sleep(delay)

For chat completions, 30–60s total budget is typical. For embedding batch jobs, you can afford larger budgets because the caller is asynchronous.

Step 6: Make requests idempotent and tagged

LLM inference is stateless, but if you pass tools or functions that mutate external state, attach an idempotency_key (UUID) and have downstream services dedupe. OpenAI-compatible APIs accept user or custom headers; forward a client-generated X-Request-Id.

import uuid

def make_headers():
    return {"X-Request-Id": str(uuid.uuid4())}

When a retry happens, reuse the same id so the gateway or provider can short-circuit duplicate billing. n4n.ai honors client routing directives and per-token metering, so stable IDs also make usage dashboards coherent instead of showing three charges for one logical call.

Step 7: Assemble a real OpenAI-compatible client wrapper

Below is a complete, runnable wrapper using the openai Python SDK pointed at any OpenAI-compatible endpoint. It applies the steps above and is explicit rather than clever.

import os, time, random, uuid
from openai import OpenAI
from openai import APIConnectionError, RateLimitError, APIStatusError

client = OpenAI(
    base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
    api_key=os.environ["LLM_API_KEY"],
)

def chat_with_retries(messages, model="gpt-4o-mini", max_attempts=5, budget=45.0):
    start = time.monotonic()
    req_id = str(uuid.uuid4())
    for attempt in range(max_attempts):
        if time.monotonic() - start > budget:
            raise TimeoutError("budget exhausted")
        try:
            return client.chat.completions.create(
                model=model,
                messages=messages,
                extra_headers={"X-Request-Id": req_id},
            )
        except RateLimitError as e:
            headers = e.response.headers if e.response else {}
            retry_after = float(headers.get("Retry-After", 0))
            raw = 0.5 * (2 ** attempt)
            delay = max(raw / 2 + random.uniform(0, raw / 2), retry_after)
            if time.monotonic() + delay - start > budget:
                raise
            time.sleep(delay)
        except APIConnectionError:
            raw = 0.5 * (2 ** attempt)
            time.sleep(raw / 2 + random.uniform(0, raw / 2))
        except APIStatusError as e:
            if e.status_code in {500, 502, 503, 504}:
                raw = 0.5 * (2 ** attempt)
                time.sleep(raw / 2 + random.uniform(0, raw / 2))
            else:
                raise
    raise RuntimeError("retries exhausted")

In production, extract the delay logic into a tenacity retry decorator with wait_func and stop_func. The explicit version above is for clarity and easy unit testing.

Step 8: Verify your exponential backoff llm api retries work

You cannot claim resilience without testing. Stand up a local fault-injecting proxy that returns 429 with Retry-After: 1 for the first three requests, then 200.

python -m pip install flask
cat > mock.py <<'EOF'
from flask import Flask, jsonify, make_response
app = Flask(__name__)
state = {"n": 0}
@app.route("/v1/chat/completions", methods=["POST"])
def comp():
    if state["n"] < 3:
        state["n"] += 1
        r = make_response(jsonify({"error": "rate limited"}), 429)
        r.headers["Retry-After"] = "1"
        return r
    return jsonify({"ok": True})
app.run(port=5000)
EOF
python mock.py

Point LLM_BASE_URL=http://localhost:5000/v1 at the mock and call chat_with_retries. Successful verification shows:

• Logs print increasing delays (0.5s, ~1s, ~2s) capped or extended by Retry-After.
• The function returns the {"ok": true} payload after three retries.
• Total elapsed time stays under budget.
• A unit test with unittest.mock patches client.chat.completions.create to raise RateLimitError and asserts time.sleep was called with correct bounds.

Add a metric counter for retry_total and retry_exhausted so you can alert when backoff frequency crosses a threshold. If you skip the fault-injection test, your exponential backoff llm api retries are theater. Ship the tests alongside the code, and your LLM integrations will survive provider hiccups without taking down your own stack.