Structured error handling for multi-provider LLM routing

Most teams treat multi-provider llm error handling as an afterthought: a single try/except around the API call and a generic 500 to the user. That approach falls apart the moment one vendor throttles you or another returns a malformed streaming chunk. Building a deliberate error taxonomy and fallback path is what separates a demo from a system that survives a provider incident.

1. Classify errors before you react

Not every non-200 is retryable. Start by splitting responses into four buckets:

• Client errors (4xx except 408/429): bad request, auth failure, content filter. No point retrying with identical args.
• Rate limits (429, sometimes 403 with retry-after): retry with backoff, possibly route elsewhere.
• Transient server errors (500, 502, 503, 504, 408): retry or fall back.
• Streaming mid-flight failures: connection dropped after first token.

Define a small exception hierarchy in your client wrapper:

class LLMError(Exception):
    retryable: bool = False
    fallback: bool = False

class RateLimitError(LLMError):
    retryable = True
    fallback = True

class ServerError(LLMError):
    retryable = True
    fallback = True

class ClientError(LLMError):
    retryable = False
    fallback = False

class StreamInterrupted(LLMError):
    retryable = False
    fallback = True  # only if no side effects

This typing drives every later decision. A ClientError should never trigger a fallback to a different provider because the prompt is unlikely to suddenly become valid.

2. Define a retry budget per request

Unbounded retries amplify load during incidents. Use a deadline and a max attempt count, and share the budget across providers.

import time

def with_budget(max_ms=2000, max_attempts=3):
    deadline = time.monotonic() + max_ms / 1000
    attempts = 0
    def guard():
        nonlocal attempts
        attempts += 1
        if attempts > max_attempts:
            raise RuntimeError("retry budget exhausted")
        if time.monotonic() > deadline:
            raise RuntimeError("deadline exceeded")
    return guard

Call guard() at the top of each attempt. If you are using a gateway that provides automatic fallback when a provider is rate-limited or degraded, you can shrink your client retry count—but you still need the budget to avoid hanging on a slow tail.

Pitfall: retrying on cached failures

If your request hit a provider cache and returned a 400, retrying against another provider wastes tokens. Honor the cache-control hints forwarded by some gateways; a cached rejection is still a rejection.

3. Build an explicit fallback chain

A fallback chain is an ordered list of (provider, model) pairs. On a fallback-flagged error, pop the next entry. Keep the original request payload identical unless the model requires format changes.

PROVIDERS = [
    ("openai", "gpt-4o-mini"),
    ("anthropic", "claude-3-haiku"),
    ("mistral", "mistral-small"),
]

def route_with_fallback(messages, chain=PROVIDERS):
    last_err = None
    for provider, model in chain:
        try:
            return call_provider(provider, model, messages)
        except (RateLimitError, ServerError, StreamInterrupted) as e:
            last_err = e
            continue
    raise last_err or RuntimeError("no providers configured")

Tradeoff: fallback changes latency and cost profile. A cheap primary that fails fast is better than a premium model as primary “just in case”. Measure p95 latency per hop.

If you use a unified endpoint such as n4n.ai, which offers automatic fallback when a provider is rate-limited or degraded, you can delegate the chain to the gateway and simply handle the normalized error it returns. Your code stays lean, but you lose fine-grained control over which model substitutes for which.

4. Normalize provider status codes

Every vendor encodes errors differently. OpenAI returns error.type; Anthropic uses error.type with status; some bare metal endpoints return raw HTTP. Map them to your taxonomy at the edge of your client.

{
  "error": {
    "message": "Rate limit reached",
    "type": "rate_limit_error",
    "code": "429",
    "provider": "openai"
  }
}

A normalization function:

def normalize(status, body, provider):
    if status == 429:
        return RateLimitError(f"{provider}: {body.get('error',{}).get('message')}")
    if status >= 500 or status in (408, 502, 503, 504):
        return ServerError(f"{provider}: {status}")
    if status == 400:
        return ClientError(f"{provider}: bad request")
    # ...

Do not trust message strings for logic. Use status codes and documented type fields.

Common pitfall: 402 Payment Required

Some providers return 402 when credits are exhausted. Treat as ClientError for that provider but flag for fallback if another provider has balance. Don’t retry the same billing account.

5. Surface partial failures to callers

Streaming breaks the request/response metaphor. If you get three tokens then a TCP reset, the user already saw output. Two patterns:

1. Buffer and validate: accumulate tokens, only flush to user after chunk N. If error before N, retry silently.
2. Mark incomplete: send a final [DONE] with an error field, let UI show “response truncated”.

type StreamEvent =
  | { type: "token"; value: string }
  | { type: "error"; retryable: boolean; message: string }
  | { type: "done" };

function emitTruncated(err: StreamInterrupted) {
  return { type: "error", retryable: false, message: "stream interrupted" } as StreamEvent;
}

Pick based on product tolerance. A coding assistant can’t silently drop a function body; a chat toy can.

6. Instrument and meter what actually happened

You cannot tune multi-provider llm error handling without data. Log every attempt: provider, model, status, latency, whether fallback triggered, and final outcome. If your gateway does per-token usage metering, correlate its records with your attempt logs to find silent retries that double-billed.

import logging

logger = logging.getLogger("llm.router")

def log_attempt(provider, model, ok, err=None, ms=0):
    logger.info("llm_attempt", extra={
        "provider": provider, "model": model,
        "ok": ok, "err": type(err).__name__, "ms": ms
    })

Dashboards should show error rate per provider, not just aggregate. A provider with 5% 503 but 200ms latency may beat a 99% reliable one at 2s.

Tradeoff: head-of-line blocking

If you serialize fallback (try A, then B), a slow A increases tail latency. Consider racing two providers on the first token, but only if cost permits. Racing complicates metering and caching.

7. Honor client routing directives

When you pass a request through a gateway, you may specify route hints or cache directives. If you ignore them client-side, you undermine the gateway’s ability to cache prefixes or respect regional rules. Forward cache-control: max-age=3600 when your prompt is static; otherwise a retry may bypass a warm cache.

curl https://api.example.com/v1/chat/completions \
  -H "authorization: bearer $KEY" \
  -H "cache-control: max-age=300" \
  -d '{"model":"gpt-4o","messages":[...]}'

The same header should accompany retries to the same provider.

Closing checklist

• Typed errors before any network call.
• Retry budget shared across providers.
• Explicit fallback order, cheapest first.
• Normalize status at client edge.
• Handle streaming truncation explicitly.
• Log per-attempt outcomes, not just failures.

Multi-provider llm error handling is not glamorous, but it is the difference between a status page fire and a silent reroute. Implement the taxonomy once, then let the routing logic stay boring.