Building a retry-with-backoff wrapper for LLM REST calls

A reliable python retry backoff llm rest api wrapper separates prototype scripts from production systems. When you call inference endpoints directly, transient 429 rate limits and 5xx upstream errors are guaranteed, and a single unhandled failure drops a user request. This tutorial builds a minimal, dependency-light wrapper around httpx that retries idempotent completion calls with exponential backoff and jitter, then tests it against a simulated flaky server.

Prerequisites

• Python 3.10 or newer
• httpx installed (pip install httpx==0.27.*)
• Familiarity with OpenAI-compatible JSON request/response shapes
• pytest if you want to run the mock test (optional)

python -m venv .venv && source .venv/bin/activate
pip install httpx pytest

Failure modes in LLM REST calls

Most LLM providers expose an OpenAI-compatible /v1/chat/completions endpoint. The errors you must survive:

• 429 Too Many Requests — rate limit or concurrency cap.
• 500/502/503/504 — provider-side degradation.
• httpx.ConnectError, httpx.ReadTimeout — network blips.

A correct python retry backoff llm rest api client treats these as retryable, but never retries 400 (bad request) or 401 (auth) because resending yields the same failure.

Step 1: Bare POST call

Start with a plain function that sends a chat completion request.

import httpx

def call_llm(base_url: str, api_key: str, model: str, prompt: str) -> dict:
    url = f"{base_url.rstrip('/')}/v1/chat/completions"
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    }
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 128,
    }
    resp = httpx.post(url, headers=headers, json=payload, timeout=30.0)
    resp.raise_for_status()
    return resp.json()

This throws httpx.HTTPStatusError on 4xx/5xx and httpx.TransportError on network issues. No resilience yet.

Step 2: Naive retry loop

A first pass might loop a fixed number of times with a sleep.

import time

def call_llm_retry_naive(base_url, api_key, model, prompt, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return call_llm(base_url, api_key, model, prompt)
        except (httpx.HTTPStatusError, httpx.TransportError) as e:
            if attempt == max_attempts - 1:
                raise
            time.sleep(1.0)  # fixed delay

Fixed delays are wrong: they hammer the server in lockstep and amplify thundering herds. Use exponential backoff.

Step 3: Exponential backoff with jitter

The delay for attempt n (0-indexed) should be base * 2**n plus random jitter. Cap the max delay.

import random

def backoff_delay(attempt: int, base: float = 0.5, cap: float = 30.0) -> float:
    """Exponential backoff with full jitter."""
    delay = min(cap, base * (2 ** attempt))
    return random.uniform(0, delay)

random.uniform(0, delay) implements “full jitter” (AWS style), which avoids synchronized retries across many clients.

Step 4: The full retry wrapper

Now combine the loop, status filtering, and backoff. We also honor Retry-After if present.

import logging
import time
import httpx

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("llm_client")

def call_llm_with_retry(
    base_url: str,
    api_key: str,
    model: str,
    prompt: str,
    max_retries: int = 5,
    base_delay: float = 0.5,
) -> dict:
    url = f"{base_url.rstrip('/')}/v1/chat/completions"
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    payload = {"model": model, "messages": [{"role": "user", "content": prompt}], "max_tokens": 128}

    for attempt in range(max_retries):
        try:
            resp = httpx.post(url, headers=headers, json=payload, timeout=30.0)
            if resp.status_code == 429:
                raise httpx.HTTPStatusError("rate limited", request=resp.request, response=resp)
            resp.raise_for_status()
            return resp.json()
        except httpx.HTTPStatusError as e:
            status = e.response.status_code if e.response else None
            # Only retry 429, 502, 503, 504
            if status not in (429, 502, 503, 504) or attempt == max_retries - 1:
                raise
            retry_after = e.response.headers.get("Retry-After")
            if retry_after and retry_after.isdigit():
                delay = float(retry_after)
            else:
                delay = backoff_delay(attempt, base_delay)
            logger.warning("Attempt %d failed (%s). Retrying in %.2fs", attempt, status, delay)
            time.sleep(delay)
        except httpx.TransportError as e:
            if attempt == max_retries - 1:
                raise
            delay = backoff_delay(attempt, base_delay)
            logger.warning("Attempt %d transport error: %s. Retrying in %.2fs", attempt, e, delay)
            time.sleep(delay)
    raise RuntimeError("Unreachable")

This python retry backoff llm rest api wrapper logs each retry and respects server hints.

Expected log output on a flaky run:

WARNING:llm_client:Attempt 0 failed (503). Retrying in 0.32s
WARNING:llm_client:Attempt 1 failed (429). Retrying in 0.91s
{'id': 'chatcmpl-123', 'choices': [{'message': {'content': 'ok'}}]}

Step 5: Test with a mock transport

httpx lets you inject a MockTransport to simulate failures without network calls.

import httpx
import pytest

def test_retry_then_success():
    calls = {"n": 0}
    def handler(request):
        calls["n"] += 1
        if calls["n"] < 3:
            return httpx.Response(503, headers={"Retry-After": "0"})
        return httpx.Response(200, json={"ok": True})

    transport = httpx.MockTransport(handler)
    with httpx.Client(transport=transport) as client:
        # monkeypatch client into call_llm_with_retry via injection would be needed;
        # for brevity, replicate logic with client param:
        resp = client.post("http://test/v1/chat/completions", json={})
        # In real code, pass client to wrapper.
        assert resp.status_code == 200
        assert calls["n"] == 3

If you refactor call_llm_with_retry to accept an httpx.Client argument, the same mock works end-to-end. That proves the backoff logic without hitting a real provider.

Step 6: Non-retryable cases

Never retry these:

• 400 malformed request — your payload is wrong.
• 401/403 auth failure — key is invalid.
• 404 wrong endpoint.
• 422 validation error.

Also, if the response streams partial tokens, retrying mid-stream is unsafe; only retry before the first byte arrives. For non-streaming JSON, the wrapper above is safe.

Gateway alternative

Rolling your own python retry backoff llm rest api layer is instructive, but a gateway such as n4n.ai handles automatic fallback when a provider is rate-limited or degraded, and forwards provider cache-control hints without extra code. Use the wrapper when you must talk to a single provider directly; use the gateway when you want multi-provider resilience.

Async variant

Production services often use asyncio. The same algorithm maps to async with httpx.AsyncClient() as client: and await asyncio.sleep(delay). Avoid blocking the event loop with time.sleep.

import asyncio, httpx

async def call_llm_async_retry(base_url, api_key, model, prompt, max_retries=5):
    # ... same logic with await client.post and await asyncio.sleep

Final checklist

• Cap total retry time (max_retries * cap) to avoid hanging requests.
• Set timeout on httpx to fail fast on dead connections.
• Log attempt number and status for observability.
• Pass a single httpx.Client for connection pooling.

That’s a complete, runnable pattern you can drop into a service today.