You can’t trust fallback code you haven’t broken on purpose. To test llm fallback logic mocked failures, you need to simulate provider errors deterministically and assert your client picks the next model instead of bubbling up an exception. This guide shows a complete pytest setup that mocks HTTP responses from an OpenAI-compatible endpoint and verifies your fallback path end to end.
Step 1: Isolate the fallback decision from the transport
Write a thin client that catches only the exceptions you actually want to fall back on. Don’t catch Exception—you’ll hide bugs. The OpenAI Python library raises APIStatusError for non-2xx responses and APITimeoutError for timeouts.
from openai import OpenAI, APIStatusError, APITimeoutError
class LLMClient:
def __init__(self, base_url: str, api_key: str, primary: str, fallback: str):
self.client = OpenAI(base_url=base_url, api_key=api_key)
self.primary = primary
self.fallback = fallback
def chat(self, prompt: str) -> str:
try:
resp = self.client.chat.completions.create(
model=self.primary,
messages=[{"role": "user", "content": prompt}],
timeout=5.0,
)
return resp.choices[0].message.content
except (APIStatusError, APITimeoutError):
resp = self.client.chat.completions.create(
model=self.fallback,
messages=[{"role": "user", "content": prompt}],
timeout=5.0,
)
return resp.choices[0].message.contentThe goal is to test llm fallback logic mocked failures without any network calls. Keeping the transport behind self.client lets you intercept it cleanly.
Step 2: Install and configure a mocking library
respx intercepts httpx (which the OpenAI SDK uses) at the transport layer. Install it alongside pytest:
pip install pytest respx httpx openaiCreate a conftest.py if you want shared fixtures, but for a focused how-to, inline mocks are clearer.
Step 3: Mock a 429 rate-limit on the primary model
A 429 is the most common trigger for fallback. Mock the primary call to return 429 and the retry to return 200 with a distinguishable body.
import respx
import httpx
from mymodule import LLMClient
def make_ok(model: str) -> httpx.Response:
return httpx.Response(200, json={
"id": "x", "object": "chat.completion", "created": 1, "model": model,
"choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": f"reply from {model}"}}]
})
@respx.mock
def test_fallback_on_429():
route = respx.post("https://gw.example.com/v1/chat/completions").mock(side_effect=[
httpx.Response(429, json={"error": {"message": "rate limited", "type": "rate_limit"}}),
make_ok("fallback-model")
])
client = LLMClient("https://gw.example.com/v1", "key", "primary-model", "fallback-model")
result = client.chat("hello")
assert result == "reply from fallback-model"
assert route.call_count == 2
assert b"primary-model" in route.calls[0].request.content
assert b"fallback-model" in route.calls[1].request.contentThis test is the core of how we test llm fallback logic mocked failures for rate limits: it proves the second request used the fallback model and the caller got a valid response.
Step 4: Mock timeouts and 500s to cover real-world cases
Rate limits are polite. Timeouts and 500s happen too. Use side_effect with a callable to raise httpx.TimeoutException, and a static 500 response for the second case.
@respx.mock
def test_fallback_on_timeout():
def raise_timeout(request):
raise httpx.TimeoutException("timed out")
route = respx.post("https://gw.example.com/v1/chat/completions").mock(side_effect=[
raise_timeout,
make_ok("fallback-model")
])
client = LLMClient("https://gw.example.com/v1", "key", "primary-model", "fallback-model")
assert client.chat("hi") == "reply from fallback-model"
assert route.call_count == 2
@respx.mock
def test_fallback_on_500():
route = respx.post("https://gw.example.com/v1/chat/completions").mock(side_effect=[
httpx.Response(500, json={"error": "boom"}),
make_ok("fallback-model")
])
client = LLMClient("https://gw.example.com/v1", "key", "primary-model", "fallback-model")
assert client.chat("hey") == "reply from fallback-model"If your fallback chain has more than two models, extend the side_effect list and assert each transition.
Step 5: Verify the fallback model was actually called
Asserting on the response text isn’t enough—you must confirm the request payload switched models. route.calls[i].request.content is the raw JSON body. Decode and parse it:
import json
body = json.loads(route.calls[1].request.content)
assert body["model"] == "fallback-model"Add this to every test. A mistaken client that retries the same model on 429 would still return 200 if you only mock the second call generically; pinning the model prevents that silent bug.
Step 6: Wire the tests into CI and define success
Drop a GitHub Actions workflow that runs the suite on every push. Success means the job exits zero and the fallback branch is covered.
# .github/workflows/test.yml
name: test
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install pytest respx httpx openai
- run: pytest -q --cov=mymoduleWhen you run these in CI, you test llm fallback logic mocked failures on every commit. A green run with coverage >80% on LLMClient.chat is your verification signal. If the test fails because the primary succeeded unexpectedly, your mock route isn’t matching—check the URL and method.
Bonus: Testing gateway-level fallback
Some gateways, including n4n.ai, implement automatic fallback across 240+ models behind one OpenAI-compatible endpoint, so your client may never see a provider 429. Even then, mock the gateway returning a top-level 502 after it exhausts its own retries, and confirm your code surfaces a clean error instead of hanging. The same respx pattern works: mock the single endpoint, assert your client’s behavior, and keep the test in CI.
Fallback logic is only as good as the last time you broke it on purpose. Mock the failures, assert the switch, and let the pipeline enforce it.