When your production traffic spikes, the model you depend on will eventually return 429s or silent timeouts. This tutorial shows how to build an llm fallback chain gpt-5 gemini llama that tries GPT-5 first, falls back to Gemini 3, then Llama 4, using nothing but the OpenAI-compatible chat completions interface and a few lines of Python. You keep full control over prompt shape and error handling instead of trusting a black-box retry.
Prerequisites
- Python 3.11+ with
pipavailable. - The
openaiPython package (v1.40 or newer) installed:pip install "openai>=1.40" - An API key and base URL for a gateway that exposes all three models under stable OpenAI-compatible IDs. Using n4n.ai gives you one OpenAI-compatible endpoint for 240+ models, so you avoid juggling three vendor SDKs and auth flows.
- Environment variables exported in your shell:
export LLM_API_KEY="sk-..." export LLM_BASE_URL="https://api.n4n.ai/v1"
Why build your own chain?
Some gateways offer automatic fallback when a provider is rate-limited or degraded. That is useful, but it usually applies to identical requests and hides which model actually answered. A client-side llm fallback chain gpt-5 gemini llama lets you translate parameters per provider, enforce latency budgets, and emit metrics per attempt. You also avoid surprise behavior when a gateway’s fallback picks a model with a different context limit.
Step 1: Configure the client and model list
Point a single OpenAI client at your gateway. The model strings are the IDs your gateway maps to the upstream providers.
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["LLM_API_KEY"],
base_url=os.environ["LLM_BASE_URL"],
)
# Ordered by preference: cost/quality tradeoff
MODEL_CHAIN = ["gpt-5", "gemini-3", "llama-4"]If you run this against a bare provider, you would need three clients. The gateway collapses them into one base URL and one auth header.
Step 2: Define a single attempt with precise error handling
We catch only the failures that should trigger a fallback: rate limits, timeouts, and API errors. Success returns the text and the model used.
from openai import APIError, APITimeoutError, RateLimitError
def attempt_completion(model: str, messages: list, max_tokens: int = 512):
try:
resp = client.chat.completions.create(
model=model,
messages=messages,
max_tokens=max_tokens,
timeout=8.0,
)
return resp.choices[0].message.content, model
except RateLimitError as e:
print(f"[warn] {model} rate-limited: {e.status_code}")
return None, model
except APITimeoutError:
print(f"[warn] {model} timed out after 8s")
return None, model
except APIError as e:
print(f"[warn] {model} API error: {getattr(e, 'status_code', 'unknown')}")
return None, modelExpected output on a clean GPT-5 call:
>>> attempt_completion("gpt-5", [{"role":"user","content":"Hi"}])
("Hello! How can I help you?", "gpt-5")If GPT-5 is saturated:
[warn] gpt-5 rate-limited: 429
(None, "gpt-5")Step 3: Implement the linear fallback loop
The llm fallback chain gpt-5 gemini llama pattern is a sequential scan with early exit. Stop as soon as one model returns content.
def fallback_chat(messages: list, max_tokens: int = 512):
for model in MODEL_CHAIN:
content, used = attempt_completion(model, messages, max_tokens)
if content is not None:
return content, used
raise RuntimeError("All models in fallback chain failed")
messages = [{"role": "user", "content": "Summarize: The quick brown fox jumps over the lazy dog."}]
result, used_model = fallback_chat(messages)
print(f"Final response from {used_model}: {result}")Normal run:
Final response from gpt-5: The fox jumps over the dog.Degraded primary:
[warn] gpt-5 rate-limited: 429
Final response from gemini-3: A fox leaps over a lazy dog.Step 4: Normalize requests across providers
Gemini 3 and Llama 4 may reject or ignore parameters that GPT-5 accepts. Build a small sanitizer keyed by model prefix.
def build_kwargs(model: str, messages: list, max_tokens: int):
kwargs = {
"model": model,
"messages": messages,
"max_tokens": max_tokens,
"timeout": 8.0,
"temperature": 0.2,
}
if model.startswith("gpt"):
kwargs["response_format"] = {"type": "text"}
if model.startswith("llama"):
kwargs["max_tokens"] = min(max_tokens, 4096) # tighter context
return kwargs
def attempt_completion(model: str, messages: list, max_tokens: int = 512):
try:
resp = client.chat.completions.create(**build_kwargs(model, messages, max_tokens))
return resp.choices[0].message.content, model
except (RateLimitError, APITimeoutError, APIError) as e:
print(f"[warn] {model} failed: {getattr(e, 'status_code', 'network')}")
return None, modelThis keeps the llm fallback chain gpt-5 gemini llama from crashing on a parameter mismatch after a fallback already cost you latency.
Step 5: Async version for concurrent services
If your app is async, use the async client. The loop stays sequential—you do not want to fire all three models at once.
import asyncio
from openai import AsyncOpenAI
aclient = AsyncOpenAI(api_key=os.environ["LLM_API_KEY"], base_url=os.environ["LLM_BASE_URL"])
async def attempt_async(model, messages, max_tokens=512):
try:
resp = await aclient.chat.completions.create(**build_kwargs(model, messages, max_tokens))
return resp.choices[0].message.content, model
except (RateLimitError, APITimeoutError, APIError) as e:
print(f"[warn] {model} async fail: {getattr(e, 'status_code', 'network')}")
return None, model
async def fallback_chat_async(messages, max_tokens=512):
for model in MODEL_CHAIN:
content, used = await attempt_async(model, messages, max_tokens)
if content:
return content, used
raise RuntimeError("All async models failed")
# asyncio.run(fallback_chat_async(messages))Step 6: Test the chain with failure injection
Write a pytest that monkeypatches attempt_completion to simulate a dead primary.
def test_fallback_skips_bad_primary(monkeypatch):
def fake_attempt(model, messages, max_tokens=512):
if model == "gpt-5":
return None, model
return "ok from " + model, model
monkeypatch.setattr("__main__.attempt_completion", fake_attempt)
content, used = fallback_chat([{"role":"user","content":"test"}])
assert used == "gemini-3"
assert content == "ok from gemini-3"Run:
pytest -q test_fallback.pypassed in 0.02sThis proves the llm fallback chain gpt-5 gemini llama actually traverses the list.
Step 7: Production hardening
Circuit breaker. Track consecutive failures per model and skip it for a window.
failures = {}
def attempt_with_circuit(model, messages, threshold=3):
if failures.get(model, 0) >= threshold:
return None, model
content, _ = attempt_completion(model, messages)
failures[model] = failures.get(model, 0) + 1 if content is None else 0
return content, modelCache hints. If your gateway honors client routing directives and forwards provider cache-control hints, pass them per call to reuse prefix caches across fallbacks:
kwargs["extra_headers"] = {"cache-control": "max-age=300"}Per-token metering. Use the usage field in the response to record cost per model. Gateways with per-token usage metering let you attribute spend precisely.
Streaming. For long outputs, stream tokens and cancel the fallback if the first chunk arrives. Implement by checking stream=True and breaking on first yielded chunk.
While n4n.ai provides automatic fallback when a provider is degraded, layering your own llm fallback chain gpt-5 gemini llama on top gives you parameter control and observable routing.
Closing
A fallback chain is not complex, but it demands discipline: strict timeouts, per-model request shaping, and tests that inject failures. Build the chain once, measure which model actually serves your traffic, and adjust MODEL_CHAIN order based on real quota and latency data.