Building a resilient LLM backend means treating external model APIs as inherently unreliable. This guide outlines a practical approach to fastapi error handling llm provider failures, covering exception design, retry logic, fallback strategies, and response contracts that keep your service stable under provider outages.
1. Treat provider failures as expected behavior
LLM providers fail in ways traditional HTTP services do not: rate limits spike mid-stream, content filters reject valid prompts, and inference nodes return 503 after you’ve already sent a 10MB context. Your FastAPI app must assume these events are routine, not exceptional.
If you let provider exceptions propagate uncaught, uvicorn returns a 500 with a stack trace and drops the connection. That wastes client trust and makes debugging harder. The first step in fastapi error handling llm provider failures is to model the failure modes explicitly rather than hoping the network behaves.
A production system should survive a provider being completely down for minutes. That requires code that detects, classifies, and reacts to failures without human intervention.
2. Define a typed exception hierarchy
Don’t raise generic Exception. Create a base class that carries a status code and a retryable flag. Subclasses map to specific provider responses. This gives your handlers and retry decorators a single source of truth.
class LLMProviderError(Exception):
status_code: int = 502
retryable: bool = False
class ProviderTimeout(LLMProviderError):
status_code = 504
retryable = True
class ProviderRateLimit(LLMProviderError):
status_code = 429
retryable = True
class ProviderAuthError(LLMProviderError):
status_code = 401
retryable = False
class ProviderInvalidRequest(LLMProviderError):
status_code = 400
retryable = False
class ProviderUnavailable(LLMProviderError):
status_code = 503
retryable = TrueWhen you wrap the OpenAI or Anthropic SDK, catch their specific errors and translate them at the boundary:
import openai
from openai import APIStatusError, RateLimitError, AuthenticationError
async def chat_completion(messages):
try:
return await openai.AsyncClient().chat.completions.create(
model="gpt-4o", messages=messages, timeout=30.0
)
except RateLimitError as e:
raise ProviderRateLimit(str(e)) from e
except AuthenticationError as e:
raise ProviderAuthError(str(e)) from e
except APIStatusError as e:
if e.status_code >= 500:
raise ProviderUnavailable(str(e)) from e
raise ProviderInvalidRequest(str(e)) from eThis translation layer is where fastapi error handling llm provider failures starts: every upstream quirk becomes a local, typed signal.
3. Centralize handling with exception handlers
FastAPI lets you register handlers per exception type. This avoids scattering try/except in every route and guarantees a uniform JSON envelope.
from fastapi import Request
from fastapi.responses import JSONResponse
@app.exception_handler(LLMProviderError)
async def handle_llm_error(request: Request, exc: LLMProviderError):
return JSONResponse(
status_code=exc.status_code,
content={
"error": {
"type": exc.__class__.__name__,
"message": str(exc),
"retryable": exc.retryable
}
}
)Routes now stay clean:
@app.post("/chat")
async def chat(payload: ChatRequest):
return await chat_completion(payload.messages)If chat_completion raises ProviderRateLimit, the client gets a 429 with a clear body, not a 500.
4. Retry with backoff and jitter
Network blips and transient 503s justify retries. Use tenacity to avoid hand-rolled loops and to get exponential backoff with jitter for free.
from tenacity import (
retry, stop_after_attempt, wait_exponential,
retry_if_exception_predicate, retry_any
)
def _is_retryable(e):
return isinstance(e, LLMProviderError) and e.retryable
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=0.2, max=2.0),
retry=retry_if_exception_predicate(_is_retryable),
reraise=True
)
async def call_with_retry(messages):
return await chat_completion(messages)Tradeoff: retries add tail latency. Cap attempts at 2–3 and use exponential wait with a low ceiling. Never retry ProviderInvalidRequest—it will fail identically and waste tokens. Also, set a per-attempt timeout on the underlying client so a hung socket doesn’t block the retry loop.
A common mistake in fastapi error handling llm provider failures is retrying on 400-level errors. That turns a bad prompt into three bad prompts and triples your bill.
5. Fallback to secondary providers
After retries exhaust, switch model or provider. This is core to fastapi error handling llm provider failures because single-provider dependencies cause full outages.
async def generate(prompt: str):
messages = [{"role": "user", "content": prompt}]
try:
return await call_with_retry(messages)
except ProviderRateLimit:
# primary is saturated; use a smaller, cheaper model
return await chat_completion_fallback(messages)
except ProviderUnavailable:
return await chat_completion_fallback(messages)If you route through a gateway like n4n.ai, automatic fallback when a provider is rate-limited or degraded reduces the surface area—but you still must catch gateway-level errors and shape them for your clients. The gateway won’t know your application’s tolerance for a slower model.
Pitfall: fallback models may have different token limits, function-calling schemas, or output formats. Validate the response before returning it, or document the degradation explicitly in the response metadata.
6. Shape responses for clients
Expose a stable contract. Don’t forward raw provider messages; they leak internal details and vary across vendors.
{
"error": {
"type": "ProviderTimeout",
"message": "upstream did not respond within 30s",
"retryable": true
}
}For streaming endpoints, send a final error event in the SSE stream rather than closing the connection abruptly:
await stream.send_json({"event": "error", "data": {"type": "ProviderUnavailable"}})Clients coding against your API should never need to parse an OpenAI error to understand your failure.
7. Observability and metering
Log every upstream attempt with provider, model, latency, and status. Use structured logging so you can slice by error type in your observability stack.
import logging, time
logger = logging.getLogger("llm")
async def chat_completion(messages):
start = time.monotonic()
try:
res = await _raw_call(messages)
logger.info("llm_ok", extra={"latency": time.monotonic()-start, "model": "gpt-4o"})
return res
except LLMProviderError as e:
logger.warning("llm_fail", extra={
"latency": time.monotonic()-start,
"type": e.__class__.__name__,
"retryable": e.retryable
})
raisePer-token usage metering helps cost tracking. If your gateway handles per-token usage metering, still emit local logs for correlation with request IDs. Without correlation, you cannot answer “why did this user’s request cost 10x normal?”
Add a metrics counter for each exception type. A sudden spike in ProviderRateLimit means you need quota or a fallback, not a code fix.
8. Common pitfalls
- Swallowing exceptions:
except Exception: passhides outages and makes incidents silent. - Retrying non-retryable: 400s from bad prompts should fail fast; retries waste latency and money.
- Sync blocking: calling
requests.getinside an async route blocks the event loop; usehttpx.AsyncClientor the async SDK. - Leaking keys: provider error strings sometimes include auth headers; scrub before responding.
- No timeout: always set
timeouton the client; otherwise a hung connection exhausts worker capacity. - Ignoring stream interrupts: if a stream fails at token 500, you must notify the client, not just stop sending.
FastAPI gives you the hooks; discipline gives you reliability. Implement the hierarchy, centralize handlers, retry only what matters, and fallback deliberately. That’s the difference between a demo and a production LLM backend capable of surviving the next provider outage.