Handling 400 vs 500 errors differently in retry logic

Most HTTP clients treat any non-2xx as a failure, but correct 400 vs 500 error retry logic separates client mistakes from server faults. Retrying a 400 just replays a bad request and hides a bug; retrying a 500 without backoff floods a degraded provider. This guide gives you a drop-in pattern for LLM inference calls that you can ship today.

Step 1: Classify HTTP status codes by retryability

Not all 4xx are equal. In an LLM gateway context, a 400 usually means malformed JSON, an unknown model name, or a parameter out of range (e.g., max_tokens too high). A 401 means your API key is missing or invalid. A 403 means the key lacks scope. A 404 means the route is wrong. A 422 means the schema validated but the values are semantically invalid. None of these will ever succeed on retry.

429 (Too Many Requests) is a 4xx but signals rate limiting. You must retry it, ideally honoring the Retry-After header. 500, 502, 503, and 504 indicate the server, upstream provider, or gateway failed and a retry may succeed.

The foundation of 400 vs 500 error retry logic is a pure function that decides what to do:

def retry_policy(status_code: int, headers: dict) -> tuple[bool, float]:
    """Return (should_retry, base_backoff_seconds)."""
    if status_code == 429:
        ra = headers.get("Retry-After")
        if ra and ra.isdigit():
            return True, float(ra)
        return True, 2.0
    if 500 <= status_code <= 599:
        return True, 1.0
    # All other 4xx: do not retry
    return False, 0.0

Keep this function side-effect free so you can unit test it exhaustively.

Step 2: Implement a status-aware retry wrapper

Use httpx for async HTTP. Wrap the post in a loop with a max attempt ceiling. Raise a typed exception on non-retryable errors so callers can catch them distinctly from network errors.

import httpx
import asyncio
import random

class NonRetryableError(Exception):
    def __init__(self, status, body):
        self.status = status
        self.body = body
        super().__init__(f"Status {status}: {body}")

async def call_llm(url, json, headers, max_attempts=4):
    attempt = 0
    while True:
        attempt += 1
        resp = await httpx.post(url, json=json, headers=headers, timeout=30)
        should_retry, wait = retry_policy(resp.status_code, resp.headers)
        if not should_retry:
            if resp.status_code >= 400:
                raise NonRetryableError(resp.status_code, resp.text)
            return resp.json()
        if attempt >= max_attempts:
            raise RuntimeError(f"Exhausted {max_attempts} attempts, last status {resp.status_code}")
        sleep = wait * (2 ** (attempt - 1)) + random.uniform(0, 0.5)
        await asyncio.sleep(sleep)

A 400 exits on the first iteration via NonRetryableError. A 503 sleeps and retries. This is the behavioral split that defines 400 vs 500 error retry logic.

Step 3: Add jitter and cap maximum backoff

Naive exponential backoff causes synchronized retries across many workers (thundering herd). Multiply the base by 2**(attempt-1), cap at 30 seconds, and add random jitter.

def backoff_time(attempt: int, base: float) -> float:
    capped = min(base * (2 ** (attempt - 1)), 30.0)
    return capped + random.uniform(0, 1.0)

Replace the sleep line in Step 2 with sleep = backoff_time(attempt, wait). For 429 with a Retry-After of 5, the first retry waits ~5s, the next (if still limited) ~10s plus jitter, etc.

Step 4: Preserve idempotency on retries

LLM completion calls are read-only but not free: they consume tokens and may produce different text. If you retry a 500 after the server processed the request but dropped the connection before responding, you could be billed twice. OpenAI-compatible endpoints accept an Idempotency-Key header to deduplicate concurrent retries.

Generate one key per logical request and send it on every attempt:

import uuid

def make_idempotency_key() -> str:
    return str(uuid.uuid4())

headers = {
    "Idempotency-Key": make_idempotency_key(),
    "Content-Type": "application/json",
}

Even a gateway that performs automatic fallback across providers—such as n4n.ai, which routes to 240+ models behind one OpenAI-compatible endpoint—will still return a 400 if your JSON schema is wrong; the idempotency key ensures a retried 500 doesn’t double-charge when the gateway retries upstream. The key must be stable across retries but unique per user intent.

Step 5: Integrate with an OpenAI-compatible client

If you use the official openai python package, inject a custom httpx client that applies the retry policy. Subclass httpx.AsyncClient and override send:

from openai import AsyncOpenAI

class RetryAsyncClient(httpx.AsyncClient):
    async def send(self, request, *args, **kwargs):
        attempt = 0
        while True:
            attempt += 1
            resp = await super().send(request, *args, **kwargs)
            should_retry, wait = retry_policy(resp.status_code, resp.headers)
            if not should_retry or attempt >= 4:
                return resp
            sleep = backoff_time(attempt, wait)
            await asyncio.sleep(sleep)

client = AsyncOpenAI(
    base_url="https://api.n4n.ai/v1",  # or your own gateway
    api_key="sk-...",
    http_client=RetryAsyncClient()
)

Now any client.chat.completions.create(...) call inherits the 400 vs 500 error retry logic automatically. Note: streaming responses complicate retries because bytes may have already been consumed; only retry stream setup failures before the first token arrives.

Step 6: Verify with fault-injecting tests

Write pytest tests that mock the HTTP layer and return a sequence of statuses. Confirm 400 raises immediately, 500 retries then succeeds, and 429 honors Retry-After.

import pytest
from unittest.mock import AsyncMock, patch

@pytest.mark.asyncio
async def test_400_no_retry():
    with patch("httpx.AsyncClient.post", new=AsyncMock()) as mock_post:
        mock_post.return_value = type("R", (), {"status_code": 400, "headers": {}, "text": "bad"})()
        with pytest.raises(NonRetryableError):
            await call_llm("url", {}, {})

@pytest.mark.asyncio
async def test_500_then_200():
    responses = [
        type("R", (), {"status_code": 503, "headers": {}, "text": "down"})(),
        type("R", (), {"status_code": 200, "headers": {}, "json": lambda: {"ok": True}})(),
    ]
    with patch("httpx.AsyncClient.post", side_effect=responses):
        result = await call_llm("url", {}, {}, max_attempts=3)
        assert result == {"ok": True}

Run pytest -q. You should see zero retries on 400 and exactly one retry on 503. For integration confidence, stand up a local stub server (e.g., pytest-httpbin or a tiny aiohttp app) that returns scripted codes, and point your client at it. Verify success by asserting the number of inbound requests the stub received and that the final parsed JSON matches expectations.

Step 7: Logging and observability

Don’t swallow the distinction. Log the status, attempt number, and whether the policy retried:

import logging
log = logging.getLogger("llm_retry")

# inside call_llm before sleep:
log.warning("retry %d for status %s after %.2fs", attempt, resp.status_code, sleep)

Emit metrics: llm_retry_total{status="500"}, llm_retry_total{status="400"}, and llm_request_nonretryable. A 400 spike is a deploy bug in your request builder; a 500 spike is provider or gateway infra. The 400 vs 500 error retry logic makes that split visible to on-call.

Common pitfalls

Treating 429 as a 400. You must retry it with backoff, not fail fast.
Retrying 401/403. That’s a credential or scope problem; fix the key, don’t loop.
Ignoring Retry-After on 429. You’ll get throttled harder.
Using one idempotency key for different prompts. Dedupe will block legitimate calls.
Retrying streaming mid-stream. Only retry before the first byte; after that, surface the error.

Follow these steps and your client will be robust against flaky providers while not masking real bugs. The 400 vs 500 error retry logic is a small piece of code with outsized impact on cost, latency, and debuggability.