LLM provider uptime benchmark: 30 days of monitoring

Most teams treat an LLM provider uptime benchmark as a single number: how often did the API return 200? Over 30 days of monitoring across major inference providers, that number proved to be the least useful metric we collected. Real reliability shows up in tail latency, rate-limit behavior, and how cleanly a provider fails when its dependency trips.

Why uptime alone misleads

A binary up/down signal hides the failure modes that actually break production LLM features. A provider can return 200 with a truncated response, a 429 that your retry loop turns into a cascading backlog, or a 503 only on requests above a certain token count. If you benchmark only “did we get a non-5xx”, you will ship a system that looks healthy in dashboards and fails silently for users.

The second problem is that LLM endpoints are not stateless compute. They sit on top of heterogeneous GPU pools,KV-cache clusters, and routing layers that shed load under contention. An LLM provider uptime benchmark that ignores request shape (model size, input tokens, concurrency) measures a different system than the one you run.

Methodology: what we actually measured

We ran a continuous probe for 30 days against six providers offering OpenAI-compatible chat endpoints. Traffic was synthetic but shaped to mirror production: 512-token average input, 256-token max output, concurrency ramped from 1 to 50 over the day.

Endpoints and traffic shape

We used a minimal async client. The point was not load testing but observing natural degradation.

import asyncio, aiohttp, time

async def probe(session, url, payload):
    try:
        async with session.post(url, json=payload, timeout=10) as resp:
            body = await resp.json()
            # success requires a valid choice, not just 200
            if "choices" in body and body["choices"]:
                return resp.status, time.time(), len(body["choices"][0]["message"]["content"])
            return resp.status, time.time(), 0
    except Exception as e:
        return 0, time.time(), 0

payload = {
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "ping"}],
    "max_tokens": 8
}

Success criteria

We scored a request as healthy only if it returned a 2xx with a non-empty choices array and latency under 30s. Anything else—timeout, 429, 500, empty completion—counted as degraded. This matches how a user-facing chatbot would experience the call.

Patterns from 30 days of monitoring

The LLM provider uptime benchmark surfaced three repeatable patterns across all vendors, though severity varied.

Hard outages vs soft degradation

Every provider had at least one multi-hour window of hard unavailability (connection refused or 503 storm). These are rare and usually publicly posted as incidents. Far more common was soft degradation: p95 latency doubling while status codes stayed green. One provider’s EU region served 200s but took 25s to first token for mid-size models during a daily peak.

Rate limits are the real uptime killer

The dominant cause of failed requests was 429, not 5xx. Providers throttle per-org TPМ/RPM in ways that are undocumented for edge cases. A burst of 20 concurrent requests often triggered limits that persisted for minutes. In an LLM provider uptime benchmark, if you count 429 as “up”, you mislead yourself. We treated it as down for the calling workload.

Regional and model-specific variance

Small models (7B–13B class) were near-invariant across regions. Frontier models showed 2–5x latency spread between US-east and Asia-pacific. One provider returned 404 for a specific snapshot model in one region while serving it fine elsewhere—a versioning drift that no global status page captured.

Building for failure: fallback and routing

Once you accept that no single provider is reliably 100%, the architecture must assume partial failure. The simplest robust pattern is ordered fallback with circuit breaking.

Code: health-check and fallback logic

async def complete_with_fallback(prompt, providers):
    last_err = None
    for p in providers:
        if p.circuit_open:
            continue
        try:
            text = await p.call(prompt)
            if text:
                return text
        except RateLimitError as e:
            p.mark_429()
            last_err = e
        except Exception as e:
            p.mark_down()
            last_err = e
    raise RuntimeError(f"all providers failed: {last_err}")

This works, but introduces tradeoffs.

Tradeoffs of automatic fallback

Fallback adds latency (serial attempts) or cost (parallel races). It can also mask a bad prompt: if provider A rejects a malformed schema and B returns garbage, you’ve traded a clean error for a silent bug. You must propagate provider metadata to the caller. Fallback also multiplies token spend during incidents because retries hit multiple bills.

What a gateway can and cannot solve

A gateway reduces operational surface area. An OpenRouter-class gateway like n4n.ai automates fallback across 240+ models and honors client routing directives, forwarding provider cache-control hints so you keep cache hits across hops. That removes the need to hand-roll the loop above.

But a gateway cannot fix a client that retries blindly. If your app sends 10 identical requests on timeout, the gateway sees 10x load and every provider behind it degrades. The LLM provider uptime benchmark we ran included a deliberate retry-storm test: naive exponential backoff without jitter took a soft event and made it a hard one.

Decisive takeaway

Stop reporting LLM provider uptime benchmark results as a percentage of 200s. Measure task success under your real traffic shape, treat 429 as failure, and design fallback as a first-class component with circuit breakers and metadata propagation. Use a gateway to handle routing, but own your retry discipline. Over 30 days, the systems that stayed usable were not on the “most uptime” provider—they were the ones engineered to degrade gracefully when the benchmark went red.