Building automatic fallback across GPT-5, Claude, and Gemini

Single-provider LLM integrations break the moment that provider hiccups. Building automatic fallback LLM providers across GPT-5, Claude, and Gemini lets your agent keep generating when one backend is rate-limited, degraded, or down. This guide walks through a concrete implementation you can ship today, with real SDK calls and a verification harness.

Step 1: Set up credentials and baseline clients

Start by installing the official SDKs. We use OpenAI’s SDK for GPT-5, Anthropic’s for Claude, and Google’s generativeai package for Gemini. Keep keys in environment variables, never hard-coded.

pip install openai anthropic google-generativeai

import os
from openai import OpenAI
from anthropic import Anthropic
import google.generativeai as genai

openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
anthropic_client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
genai.configure(api_key=os.environ["GEMINI_API_KEY"])

Each client has a different calling convention. Do not try to unify them at the HTTP layer yet—wrap them behind one Python function later. The goal of automatic fallback LLM providers is resilience, not abstraction for its own sake.

Step 2: Define a normalized message interface

Your internal code should speak one dialect. We use OpenAI’s messages list as the canonical shape: [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}].

from dataclasses import dataclass
from typing import List, Dict

Message = Dict[str, str]  # {"role": "user"|"system"|"assistant", "content": str}

def to_anthropic(messages: List[Message]):
    system = next((m["content"] for m in messages if m["role"] == "system"), None)
    conv = [{"role": m["role"], "content": m["content"]} 
            for m in messages if m["role"] != "system"]
    return system, conv

def to_gemini(messages: List[Message]):
    # Gemini uses "model" instead of "assistant"
    out = []
    for m in messages:
        role = "model" if m["role"] == "assistant" else m["role"]
        out.append({"role": role, "parts": [m["content"]]})
    return out

These converters are the only place that knows about provider quirks. Everything else stays clean.

Step 3: Implement the ordered fallback chain

The core logic tries GPT-5 first, then Claude, then Gemini. Catch provider-specific rate limit and transport errors. Wrap blocking SDK calls in asyncio.to_thread so the event loop stays free.

import asyncio
from openai import RateLimitError, APIConnectionError
from anthropic import RateLimitError as AnthropicRateLimit
from google.api_core.exceptions import ResourceExhausted, ServiceUnavailable

async def complete_with_fallback(messages: List[Message], max_tokens=1024) -> str:
    # 1. GPT-5
    try:
        resp = await asyncio.to_thread(
            openai_client.chat.completions.create,
            model="gpt-5", messages=messages, max_tokens=max_tokens
        )
        return resp.choices[0].message.content
    except (RateLimitError, APIConnectionError) as e:
        print(f"gpt-5 unavailable: {e}")

    # 2. Claude
    try:
        system, conv = to_anthropic(messages)
        kwargs = {"model": "claude-3-5-sonnet", "max_tokens": max_tokens, "messages": conv}
        if system: kwargs["system"] = system
        resp = await asyncio.to_thread(anthropic_client.messages.create, **kwargs)
        return resp.content[0].text
    except AnthropicRateLimit as e:
        print(f"claude unavailable: {e}")

    # 3. Gemini
    try:
        model = genai.GenerativeModel("gemini-1.5-pro")
        resp = await asyncio.to_thread(model.generate_content, to_gemini(messages))
        return resp.text
    except (ResourceExhausted, ServiceUnavailable) as e:
        print(f"gemini unavailable: {e}")

    raise RuntimeError("all providers exhausted")

This pattern for automatic fallback LLM providers is explicit and debuggable. You can see exactly which backend served the response.

Step 4: Handle provider-specific response limits

Claude requires max_tokens on every call; omitting it throws. Gemini’s generate_content returns a Candidate with .text but may truncate on safety filters—check resp.prompt_feedback in production. OpenAI streams by default only if you ask; we disabled it for simplicity.

If you need streaming, wrap each provider’s stream iterator separately and yield normalized deltas. Fallback mid-stream is hard; better to fail fast before the first token if the primary is obviously down (use a quick health check or circuit breaker, next step).

Step 5: Add timeouts and a circuit breaker

A hung connection is worse than a fast error. Wrap the whole chain in a timeout, and skip a provider that has failed repeatedly in the last minute.

from collections import defaultdict
import time

failures = defaultdict(lambda: 0)
cooldown = 60  # seconds

async def complete_resilient(messages, timeout=30):
    now = time.time()
    order = ["gpt-5", "claude", "gemini"]
    # crude circuit break: drop providers with >3 recent fails
    active = [p for p in order if failures[p] < 3 or now - failures[p+"_t"] > cooldown]
    for p in active:
        try:
            return await asyncio.wait_for(_call_provider(p, messages), timeout)
        except Exception as e:
            failures[p] += 1
            failures[p+"_t"] = now
    raise RuntimeError("circuit open or all failed")

_call_provider is a refactor of Step 3’s try blocks into a dispatcher. In real code, use a proper token-bucket or half-open circuit library; the snippet shows the intent.

Step 6: Verify success with a forced-failure harness

You cannot wait for a real outage to test. Monkeypatch the clients to raise, then assert the fallback returns Gemini text.

import pytest
from openai import RateLimitError

def test_fallback_chain(monkeypatch):
    def boom(*a, **k): raise RateLimitError("rate", response=None, body=None)
    monkeypatch.setattr(openai_client.chat.completions, "create", boom)
    monkeypatch.setattr(anthropic_client.messages, "create", boom)
    
    async def run():
        return await complete_with_fallback([{"role":"user","content":"hi"}], max_tokens=10)
    
    result = asyncio.run(run())
    assert isinstance(result, str) and len(result) > 0

Run pytest. If the test passes, your automatic fallback LLM providers logic works end to end. For a live check, temporarily export a bad OPENAI_API_KEY and watch the logs show gpt-5 unavailable followed by a Claude response.

Step 7: Offload the chore to a gateway

Maintaining three SDKs, converters, and circuit state is manageable but not free. If you’d rather not operate that machinery, point a single OpenAI-compatible client at n4n.ai’s endpoint. It addresses 240+ models and performs automatic fallback when a provider is rate-limited or degraded, while metering per-token usage and forwarding cache-control hints. Your code shrinks to one POST.

from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=os.environ["N4N_KEY"])
resp = client.chat.completions.create(model="gpt-5", messages=messages)

You still write the same OpenAI call; the gateway handles the cross-provider retry. That is the same resilience we built by hand, without the bespoke wrappers.

Production notes

• Latency: Sequential fallback adds tail latency. Run providers in parallel with asyncio.wait_for on the first to return if you can tolerate duplicate spend.
• Idempotency: Retries across providers can double-send side-effecting prompts. Keep generation read-only; act on the final text.
• Cost: GPT-5 and Claude price differently per token. Log which backend served each request to track spend.
• Model drift: “gpt-5” and “claude-3-5-sonnet” are placeholders for your actual pinned versions. Pin them; never use floating aliases in fallback code.

Building automatic fallback LLM providers is mostly disciplined error catching and message mapping. Do it once, test it with forced failures, and your agents will survive the next provider hiccup without a page at 3 a.m.