A fallback pattern for LLM APIs using try/except chains

When you wire multiple model providers into production, a naive single-call integration dies the moment one provider returns 429s. The try except fallback pattern llm api approach lets you chain candidates in order, catching transport and API errors and moving to the next without dropping the user request.

Prerequisites

• Python 3.11 or newer.
• openai Python SDK >= 1.30 (pip install openai).
• API keys for at least two OpenAI-compatible endpoints (e.g., OpenAI and Groq, or a gateway).
• Familiarity with Python exceptions and the chat.completions.create call shape.

No web framework required. We will use the standard library and the OpenAI client, which speaks to any OpenAI-compatible base URL. If you normally call Anthropic or Gemini, wrap them in an OpenAI-compatible proxy or use their SDK inside the same loop; the pattern is identical.

Choosing provider order and model mapping

Before writing code, decide your priority. Usually the primary is the highest-quality or lowest-cost model you trust, and fallbacks are cheaper or more available. Map model names per provider explicitly; do not assume gpt-4o-mini exists on a non-OpenAI endpoint.

PROVIDERS = [
    {
        "name": "openai",
        "base_url": "https://api.openai.com/v1",
        "api_key": "sk-your-key",
        "model": "gpt-4o-mini",
    },
    {
        "name": "groq",
        "base_url": "https://api.groq.com/openai/v1",
        "api_key": "gsk-your-key",
        "model": "llama3-8b-8192",
    },
    {
        "name": "together",
        "base_url": "https://api.together.xyz/v1",
        "api_key": "together-key",
        "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
    },
]

The try except fallback pattern llm api becomes trivial to extend: append a dict and the loop picks it up. Keep secrets in environment variables, not in source.

Step 1: Write the first try/except fallback loop

The simplest version iterates the list, attempts a completion, and on any exception prints and continues. If all fail, it re-raises the last error.

from openai import OpenAI

def complete_naive(prompt: str) -> str:
    last_err: Exception | None = None
    for p in PROVIDERS:
        try:
            client = OpenAI(base_url=p["base_url"], api_key=p["api_key"])
            resp = client.chat.completions.create(
                model=p["model"],
                messages=[{"role": "user", "content": prompt}],
                timeout=10,
            )
            return resp.choices[0].message.content
        except Exception as e:
            last_err = e
            print(f"[{p['name']}] failed: {type(e).__name__}: {e}")
    if last_err:
        raise last_err
    raise RuntimeError("no providers configured")

Run it with a prompt and both providers temporarily misconfigured:

try:
    print(complete_naive("Say hi."))
except Exception as e:
    print("All failed:", e)

Expected output when keys are invalid:

[openai] failed: AuthenticationError: Incorrect API key provided
[groq] failed: AuthenticationError: Incorrect API key provided
All failed: Incorrect API key provided

This proves the chain executes. But catching bare Exception is sloppy: a programming bug (e.g., KeyError on config) would be silently swallowed and misreported as a provider outage. The next step fixes that.

Step 2: Narrow the exception classes

The try except fallback pattern llm api should only catch errors that are retryable across providers. Auth failures and invalid request bodies should bubble immediately. The OpenAI SDK exposes specific subclasses.

from openai import (
    APIConnectionError,
    APIStatusError,
    RateLimitError,
    APITimeoutError,
)

RETRYABLE = (APIConnectionError, APITimeoutError, RateLimitError, APIStatusError)

def complete_strict(prompt: str) -> str:
    last_err: Exception | None = None
    for p in PROVIDERS:
        try:
            client = OpenAI(base_url=p["base_url"], api_key=p["api_key"])
            resp = client.chat.completions.create(
                model=p["model"],
                messages=[{"role": "user", "content": prompt}],
                timeout=10,
            )
            return resp.choices[0].message.content
        except RETRYABLE as e:
            last_err = e
            print(f"[{p['name']}] retryable failure: {e}")
            continue
        # Non-retryable (auth, bad request) propagates
    if last_err:
        raise last_err
    raise RuntimeError("no providers configured")

Now a 401 from the first provider raises instantly instead of wasting a call to the second. A 429 or 503 triggers fallback. APIStatusError covers 5xx; inspect e.status_code if you want to skip only on >=500. This is the core of a correct try except fallback pattern llm api.

Step 3: Add a single retry with backoff

Transient network blips happen. One in-process retry per provider before moving on is reasonable. We avoid external dependencies with a tiny sleep and linear backoff.

import time

def complete_with_retry(prompt: str, per_provider_retries: int = 1) -> str:
    last_err: Exception | None = None
    for p in PROVIDERS:
        for attempt in range(per_provider_retries + 1):
            try:
                client = OpenAI(base_url=p["base_url"], api_key=p["api_key"])
                resp = client.chat.completions.create(
                    model=p["model"],
                    messages=[{"role": "user", "content": prompt}],
                    timeout=10,
                )
                return resp.choices[0].message.content
            except RETRYABLE as e:
                last_err = e
                if attempt < per_provider_retries:
                    time.sleep(0.5 * (attempt + 1))
                    continue
                print(f"[{p['name']}] exhausted retries: {e}")
    if last_err:
        raise last_err
    raise RuntimeError("no providers configured")

Expected output if OpenAI hits a 429 twice but Groq succeeds:

[openai] exhausted retries: Rate limit reached (429)
Hello! How can I help you today?

Do not apply exponential backoff with long sleeps in a request path; that blocks the user. Push longer retries to a queue if needed.

Step 4: Test the chain with mocked failures

You should not depend on live outages to verify fallback. Inject fake clients so the test is deterministic and fast.

from openai import RateLimitError

class FakeResp:
    class Choice:
        message = type("M", (), {"content": "ok"})()
    choices = [Choice()]

class FakeClient:
    def __init__(self, fail=False):
        self.fail = fail
    def chat(self):
        return type("C", (), {"completions": self})()
    def create(self, **kw):
        if self.fail:
            raise RateLimitError("rate", response=None, body=None)
        return FakeResp()

def complete_injected(prompt: str, clients: list) -> str:
    last_err = None
    for name, client, model in clients:
        try:
            resp = client.chat.completions.create(
                model=model, messages=[{"role":"user","content":prompt}], timeout=10
            )
            return resp.choices[0].message.content
        except RETRYABLE as e:
            last_err = e
            print(f"[{name}] failed: {e}")
    if last_err:
        raise last_err
    raise RuntimeError("no clients")

clients = [("primary", FakeClient(fail=True), "m1"), ("secondary", FakeClient(fail=False), "m2")]
print(complete_injected("hi", clients))

Output:

[primary] failed: RateLimitError: rate
ok

This confirms the try except fallback pattern llm api works without network calls and without polluting your real keys.

Step 5: Production considerations

• Logging: Replace print with structured logging that includes provider, model, error_type, and attempt.
• Metrics: Emit a counter llm_fallback_total labeled by from/to provider to see if you are falling back too often. A fallback rate above a few percent signals a primary provider problem.
• Timeouts: Always set timeout. A hung TCP connection without timeout will block your request thread indefinitely.
• Cache-Control: If you pass extra_headers={"cache-control": "max-age=300"}, some providers honor prompt caching. Forward the same header across fallbacks to avoid recomputing system prompts.
• Token metering: Capture resp.usage and ship it to your billing system. Different providers report token counts differently; normalize before aggregating.
• Streaming: If you stream, wrap the iterator; on the first chunk error, you cannot seamlessly switch providers mid-stream. Buffer a few tokens or fall back only before the first token arrives.

When a gateway makes sense

Maintaining this loop, provider SDK updates, and per-provider model name maps is ongoing toil. If you would rather consume one OpenAI-compatible endpoint that addresses 240+ models and performs automatic fallback when a provider is rate-limited or degraded, a gateway like n4n.ai collapses the try/except chain into a single network call. You still keep the try/except for truly unexpected errors, but the cross-provider retry logic moves out of your codebase.

Final complete script

import time
from openai import OpenAI, APIConnectionError, APITimeoutError, RateLimitError, APIStatusError

PROVIDERS = [
    {"name": "openai", "base_url": "https://api.openai.com/v1", "api_key": "sk-...", "model": "gpt-4o-mini"},
    {"name": "groq", "base_url": "https://api.groq.com/openai/v1", "api_key": "gsk-...", "model": "llama3-8b-8192"},
]

RETRYABLE = (APIConnectionError, APITimeoutError, RateLimitError, APIStatusError)

def complete(prompt: str, per_provider_retries: int = 1) -> str:
    last_err: Exception | None = None
    for p in PROVIDERS:
        for attempt in range(per_provider_retries + 1):
            try:
                client = OpenAI(base_url=p["base_url"], api_key=p["api_key"])
                resp = client.chat.completions.create(
                    model=p["model"],
                    messages=[{"role": "user", "content": prompt}],
                    timeout=10,
                )
                return resp.choices[0].message.content
            except RETRYABLE as e:
                last_err = e
                if attempt < per_provider_retries:
                    time.sleep(0.5 * (attempt + 1))
                    continue
                print(f"[{p['name']}] exhausted retries: {e}")
    if last_err:
        raise last_err
    raise RuntimeError("no providers configured")

if __name__ == "__main__":
    try:
        print(complete("Say hi."))
    except Exception as e:
        print("All providers failed:", e)

The try except fallback pattern llm api is boring by design: it turns unpredictable upstream outages into a deterministic ordered list of attempts. Ship the strict exception version, add metrics, and you will sleep through the next provider incident.