Model deprecations hit production traffic without warning. Building an llm fallback chain for deprecated models is the difference between a silent 404 and a graceful migration to a supported replacement. This guide walks through a concrete implementation using OpenAI-compatible endpoints and explicit routing logic you control.
Step 1: Inventory model dependencies and deprecation signals
Start by listing every model string your services call. Pull them from config files, environment variables, and hardcoded clients. In a mature codebase the strings hide behind helper functions, so grep for model= and chat.completions.create.
grep -rn "model=" services/ --include=*.py | grep -v testMap each call site to a business function. A summarizer can tolerate more latency than an interactive chat, so its chain can be longer.
{
"services": {
"summarizer": "openai/gpt-4-0314",
"classifier": "anthropic/claude-2"
}
}Subscribe to provider deprecation notices. OpenAI and Anthropic post sunset dates months ahead, but legacy model snapshots like gpt-4-0314 disappear on a fixed schedule. Track those dates in a simple CSV or a CI check that reads a sunset.json.
A deprecated model returns 404 or 410 from the completions endpoint. Your fallback logic must treat those as terminal for that model and move on. Transient 429 or 5xx are different; those should trigger retry, not model substitution, unless the gateway already handles it.
Step 2: Define an explicit fallback priority list
Don’t rely on implicit provider upgrades. The llm fallback chain for deprecated models should be explicit, not accidental. Define a chain where each successor is semantically close enough to not break prompts.
{
"chains": {
"summarizer": [
"openai/gpt-4-0314",
"openai/gpt-4-0613",
"openai/gpt-4-turbo"
],
"classifier": [
"anthropic/claude-2",
"anthropic/claude-2.1",
"anthropic/claude-3-haiku-20240307"
]
}
}Order matters. Put the deprecated model first so you keep using it until the day it vanishes. The next entry should share the same tokenizer and response shape where possible. If you jump from Claude 2 to Haiku, expect different instruction adherence; add a prompt version tag to your chain config so you can A/B.
Document the capability matrix alongside the chain. Note max context, JSON mode support, and function calling. A fallback that drops function calling will silently break your tool-use loop.
Step 3: Implement the fallback caller
Implementing the llm fallback chain for deprecated models in code gives you auditability. Use the OpenAI Python client against any OpenAI-compatible gateway. If you point the client at n4n.ai, one OpenAI-compatible endpoint addresses 240+ models and automatically routes around rate-limited providers, but your explicit chain still owns deprecation handling.
from openai import OpenAI, APIError
import time
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-yourkey")
def complete_with_fallback(messages, model_chain, max_retries=2, **kwargs):
last_err = None
for model in model_chain:
for attempt in range(max_retries):
try:
return client.chat.completions.create(
model=model,
messages=messages,
**kwargs
)
except APIError as e:
if e.status_code in (404, 410):
last_err = e
break # deprecated, try next model
if e.status_code == 429 and attempt < max_retries - 1:
time.sleep(2 ** attempt)
continue
raise
raise last_err or RuntimeError("No models in chain")The function tries each model in order. A 404/410 triggers the next entry. A 429 retries within the same model up to max_retries before falling through. Any 5xx is re-raised; let the gateway’s own retry or your outer backoff handle transient degradation.
Set a total request timeout. A stuck connection should not block the chain for 30 seconds when a healthy fallback exists.
client.timeout = 10.0Step 4: Forward cache hints and routing directives
Prompt caching saves money and latency, but only if cache-control headers survive the fallback hop. Gateways such as n4n.ai honor client routing directives and forward provider cache-control hints, so your chain doesn’t lose caching when it switches models.
When calling Anthropic models through the compatible endpoint, pass the cache marker in the message body:
messages = [
{"role": "system", "content": "You are a strict classifier."},
{
"role": "user",
"content": "Classify: ...",
"cache_control": {"type": "ephemeral"}
}
]If your gateway supports a routing directive header, set it per attempt to pin a provider region:
extra_headers = {"x-routing-directive": "provider=anthropic"}
client.chat.completions.create(
model="anthropic/claude-2.1",
messages=messages,
extra_headers=extra_headers
)Keep the same cache_control block across fallback models. Newer models accept the same hint; older ones ignore it. Don’t recompute the prefix on fallback—reuse the exact message array so the cache key stays stable.
Step 5: Add observability and per-token metering
Log which model actually served the request. The response object exposes model and usage. Per-token metering lets you spot when traffic silently shifts to a pricier fallback.
resp = complete_with_fallback(messages, chain)
depth = chain.index(resp.model)
print(f"served={resp.model} depth={depth} tokens={resp.usage.total_tokens}")Export these fields to your metrics pipeline with a fallback_depth tag.
metrics.incr("llm.fallback.depth", tags=[f"depth:{depth}", f"service:{service}"])If depth > 0 in steady state, your primary model is gone and you need to update config. Set an alert when depth exceeds zero for more than five minutes; that’s your deprecation canary.
Step 6: Verify the chain with a forced deprecation test
Write a test that simulates the deprecated model returning 410. Monkeypatch the client method. This proves the llm fallback chain for deprecated models works before a provider does it for you.
import pytest
from openai import APIError
def test_fallback_on_deprecation(monkeypatch):
calls = []
def fake_create(*args, **kwargs):
calls.append(kwargs["model"])
if kwargs["model"] == "openai/gpt-4-0314":
raise APIError("deprecated", response=None, body=None, status_code=410)
class R:
model = kwargs["model"]
usage = type("U", (), {"total_tokens": 10})()
return R()
monkeypatch.setattr(client.chat.completions, "create", fake_create)
chain = ["openai/gpt-4-0314", "openai/gpt-4-0613"]
resp = complete_with_fallback([{"role":"user","content":"hi"}], chain)
assert resp.model == "openai/gpt-4-0613"
assert calls == chainRun it in CI on every config change. Success means the test passes and your logs show the second model served. In production, verify by temporarily pointing a canary at a known-dead model string and confirming no 404 reaches callers.
For streaming endpoints, assert that the first chunk from the fallback model arrives within your timeout. Deprecation should not cause a hung stream.
Step 7: Automate config updates before sunset
Deprecation dates are known in advance. Add a CI job that fails if today’s date exceeds the sunset minus 7 days and the deprecated string is still first in its chain.
#!/usr/bin/env python
import json, datetime
sunset = datetime.date(2024, 6, 13)
chain = json.load(open("chains.json"))["summarizer"]
if datetime.date.today() > sunset - datetime.timedelta(days=7) and "gpt-4-0314" in chain[:1]:
raise SystemExit("Primary model deprecated soon; promote fallback")This catches drift. The llm fallback chain for deprecated models is only as good as the day you remember to retire the dead entry. Once the sunset passes, remove the model from the chain entirely; leaving it first adds a useless failing call on every request.
What good looks like
After following these steps, your service handles model removal without code deploys. The fallback chain tries the preferred model, degrades to equivalents, and emits metrics that tell you when a deprecation actually happened. You stop fearing provider emails and start treating model versions as mutable configuration.
Make the chain data-driven, keep cache hints intact, and test the 410 path before it hits production. That’s the whole job.