How to stress test multi-provider LLM failover

Stress testing LLM failover is the only way to trust a multi-provider inference stack when a primary vendor starts returning 429s at 2 a.m. You need a harness that drives concurrency, injects faults, and measures whether your gateway actually shifts traffic to a healthy provider. This tutorial builds that harness from scratch using Python’s async primitives and a mock gateway, then shows how to point it at a real endpoint.

Prerequisites

• Python 3.11 or newer
• httpx for async HTTP client (pip install httpx)
• aiohttp to run a local mock gateway (pip install aiohttp)
• Comfort with asyncio and basic HTTP semantics

No external load generator required. We will write our own so we control exactly how failures are induced and observed.

Step 1: Mock a multi-provider gateway with real fallback

A real gateway fronts multiple providers and retries on a secondary when the primary degrades. We replicate that with three routes: /primary, /secondary, and the gateway entry /v1/chat/completions. The primary fails 70% of the time with a 503; the gateway catches that and calls secondary.

# gateway.py
import random
from aiohttp import web
import httpx

async def primary(request):
    if random.random() < 0.7:
        return web.Response(status=503, text="provider degraded")
    await asyncio.sleep(0.01)
    return web.json_response({"source": "primary"})

async def secondary(request):
    await asyncio.sleep(0.005)
    return web.json_response({"source": "secondary"})

async def gateway(request):
    async with httpx.AsyncClient() as c:
        try:
            r = await c.get("http://localhost:8080/primary", timeout=1.0)
            if r.status_code == 200:
                return web.json_response(await r.json(), headers={"X-Fallback": "false"})
        except Exception:
            pass
        r2 = await c.get("http://localhost:8080/secondary", timeout=1.0)
        return web.json_response(await r2.json(), headers={"X-Fallback": "true"})

async def app_factory():
    app = web.Application()
    app.router.add_get("/primary", primary)
    app.router.add_get("/secondary", secondary)
    app.router.add_post("/v1/chat/completions", gateway)
    return app

if __name__ == "__main__":
    import asyncio
    web.run_app(app_factory(), port=8080)

Run it:

python gateway.py

You now have a local endpoint that exercises failover on every request where primary is unhealthy.

Step 2: Write the stress client

We use httpx.AsyncClient with explicit connection limits. Unbounded task creation will OOM your client before it stresses the server—always cap concurrency with a semaphore.

# stress.py
import asyncio
import time
import httpx

URL = "http://localhost:8080/v1/chat/completions"
HEADERS = {"Content-Type": "application/json"}
PAYLOAD = {"model": "mock", "messages": [{"role": "user", "content": "ping"}]}

async def single_request(client, stats):
    start = time.monotonic()
    try:
        r = await client.post(URL, json=PAYLOAD, headers=HEADERS, timeout=5.0)
        stats["total"] += 1
        if r.status_code == 200:
            stats["ok"] += 1
            if r.headers.get("X-Fallback") == "true":
                stats["fallback"] += 1
        else:
            stats["error"] += 1
    except Exception:
        stats["exception"] += 1
        stats["total"] += 1
    stats["latencies"].append(time.monotonic() - start)

async def run_load(concurrency: int, total: int):
    limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency)
    stats = {"total":0,"ok":0,"error":0,"exception":0,"fallback":0,"latencies":[]}
    async with httpx.AsyncClient(limits=limits) as client:
        sem = asyncio.Semaphore(concurrency)
        async def worker():
            async with sem:
                await single_request(client, stats)
        tasks = [asyncio.create_task(worker()) for _ in range(total)]
        await asyncio.gather(*tasks)
    return stats

Step 3: Run a first pass and read output

Launch the mock gateway in one terminal, then drive 500 requests at 50 concurrent:

python -c "import asyncio, stress; import json; s=asyncio.run(stress.run_load(50,500)); s.pop('latencies'); print(json.dumps(s))"

Expected output:

{"total":500,"ok":500,"error":0,"exception":0,"fallback":347}

Every request succeeded because the gateway fell back. The fallback count near 70% of total confirms the backup path is exercised under load. If you see error or exception > 0, your client timeout or gateway retry budget is misconfigured.

Step 4: Add latency percentiles

Raw counts hide tail latency. Extend the reporter to compute p95 and p99 from the latencies list.

def report(stats):
    lat = sorted(stats["latencies"])
    n = len(lat)
    p95 = lat[int(n*0.95)] if n else 0
    p99 = lat[int(n*0.99)] if n else 0
    print(f"total={stats['total']} ok={stats['ok']} fallback={stats['fallback']} "
          f"p95={p95*1000:.1f}ms p99={p99*1000:.1f}ms")

Call report(stats) after run_load. In our mock, p95 should stay under 30ms because both providers are local. In production, a fallback usually adds network round-trips—your stress testing LLM failover harness must surface that cost.

Step 5: Ramp concurrency to find the cliff

A fixed concurrency mask hides saturation points. Ramp from 10 to 500 in steps, pausing between waves:

async def ramp():
    for conc in [10, 50, 100, 250, 500]:
        stats = await run_load(conc, conc*10)
        report(stats)
        await asyncio.sleep(2)

Run it. Watch fallback ratio stay constant while p99 climbs. If exception spikes at 250, your gateway’s upstream connection pool is exhausted—exactly the kind of defect static tests miss.

Step 6: Inject worse faults

To make stress testing LLM failover realistic, make the primary hang instead of returning 503. Edit primary:

async def primary(request):
    if random.random() < 0.7:
        await asyncio.sleep(2.0)  # hang
        return web.Response(status=503)
    return web.json_response({"source":"primary"})

Now the gateway’s client timeout (1.0s) triggers fallback. Re-run the ramp. You should see p95 jump by ~1s on fallback hits—proof that failover is not free. Tune the gateway’s timeout based on this data.

Step 7: Point the harness at a real gateway

The same client works against any OpenAI-compatible endpoint. If you run this against a real multi-provider gateway such as n4n.ai, which provides automatic fallback when a provider is rate-limited or degraded and honors client routing directives, you can pin a specific provider via its routing header and observe error rates under load. Swap the constants:

URL = "https://api.n4n.ai/v1/chat/completions"
HEADERS = {
    "Authorization": "Bearer $YOUR_KEY",
    "Content-Type": "application/json",
    # routing directive per provider docs to force a specific path
}
PAYLOAD = {
    "model": "anthropic/claude-3.5-sonnet",  # one of 240+ models
    "messages": [{"role": "user", "content": "ping"}]
}

Send a few thousand requests with a deliberately throttled API key or a provider known to be flaky in your region. Track fallback via any vendor-specific header or simply watch error stay near zero while ok holds. That is the production version of stress testing LLM failover.

What the data should tell you

• Fallback ratio under injected faults should match your configured primary failure rate. If it’s lower, retries are silently disabled.
• p99 latency during fallback should be bounded by your gateway timeout plus secondary latency. If it isn’t, your client timeout is too high.
• Exception count must be zero at configured concurrency. Transport-level errors mean the client or gateway exhausted sockets.

Don’t trust a vendor’s docs about failover. Measure it with concurrency and fault injection before you ship. The harness above is ~120 lines; there is no excuse to run blind.