n4nAI
Open console →
Benchmarks & performanceAnalysisProvider Uptime and Reliability Benchmarks

Measuring 5xx error rates across major LLM APIs

A practical analysis of how to measure 5xx error rate LLM API across major providers, why status pages mislead, and how to build resilient fallback.

n4n Team5 min read1,043 words

Audio narration

Coming soon — every post will get a voice note here.

The 5xx error rate LLM API consumers experience is the single most ignored reliability metric in generative AI stacks. While providers publish polished status pages, the truth from a client socket is messier: intermittent 503s from GPU queue saturation, 502s from load balancer restarts, and 500s from model worker crashes show up long before any incident is acknowledged.

Why provider status pages don’t tell the whole story

Most LLM providers report uptime as aggregated success of their edge routers, not end-to-end inference success. If a request hits the auth layer and passes but then fails in the model worker, the provider might count it as “served” because the API gateway responded 200 with an error payload—or they might not count it at all. From your side, you got a 500.

Worse, regional and account-tier isolation means the “global” status is irrelevant. A degraded A100 pool in us-east-1 for a specific tenant can return 503s for hours while status.openai.com shows green. I’ve seen Anthropic return 503 on claude-2 during peak EU hours while their status page stayed quiet. The 5xx error rate LLM API you care about is local to your traffic, not the provider’s marketing.

Defining and capturing 5xx correctly

Not every error is a 5xx. 429 (rate limit) and 401/403 are client-side or capacity governance, not server faults. We care about 500, 502, 503, 504. Also note that some providers tunnel errors inside 200 with error field; you must inspect the body. OpenAI sometimes returns 200 with {"error": ...} for content filter, but that’s not a 5xx. For measurement, treat any non-2xx that is not 4xx (except 429) as server fault.

A robust probe must:

  • Use the same auth and headers as production.
  • Send a minimal valid payload (small prompt, low max_tokens).
  • Record the HTTP status, latency, and response body snippet.
  • Run from the same network egress as your production calls.

Streaming complications

Many LLM calls use server-sent events. A provider may return 200 with Transfer-Encoding: chunked and then inject an error event mid-stream. Your HTTP client sees 200, but the application gets a truncated completion. For reliability measurement, you must parse the stream and treat a non-success event as a 5xx equivalent. This is why passive production logging beats simple curl.

Minimal Python harness

import asyncio, httpx, time, json

ENDPOINTS = {
    "openai": "https://api.openai.com/v1/chat/completions",
    "anthropic": "https://api.anthropic.com/v1/messages",
    "mistral": "https://api.mistral.ai/v1/chat/completions",
}

HEADERS = {
    "openai": {"Authorization": "Bearer $OPENAI_KEY", "Content-Type": "application/json"},
    "anthropic": {"x-api-key": "$ANTHROPIC_KEY", "anthropic-version": "2023-06-01", "Content-Type": "application/json"},
    "mistral": {"Authorization": "Bearer $MISTRAL_KEY", "Content-Type": "application/json"},
}

PAYLOADS = {
    "openai": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}], "max_tokens": 1},
    "anthropic": {"model": "claude-3-haiku-20240307", "max_tokens": 1, "messages": [{"role": "user", "content": "hi"}]},
    "mistral": {"model": "mistral-small-latest", "messages": [{"role": "user", "content": "hi"}], "max_tokens": 1},
}

async def probe(name, client):
    try:
        r = await client.post(ENDPOINTS[name], headers=HEADERS[name], json=PAYLOADS[name], timeout=10)
        return {"name": name, "status": r.status_code, "rt": r.elapsed.total_seconds()}
    except Exception as e:
        return {"name": name, "status": 0, "error": str(e)}

async def main():
    async with httpx.AsyncClient() as client:
        for _ in range(20):
            results = await asyncio.gather(*[probe(n, client) for n in ENDPOINTS])
            print(json.dumps(results))
            await asyncio.sleep(30)

asyncio.run(main())

This loops every 30s, 20 rounds. You’ll quickly see patterns: OpenAI tends to be 200 with occasional 503 during model hot-reloads; Mistral might throw 502 when their gateway rolls.

Interpreting 5xx subtypes in LLM land

  • 500 Internal Error: Usually a model worker crash or unhandled exception in the inference server. Rare but fatal for that request.
  • 502 Bad Gateway: Edge proxy couldn’t reach the backend; common during deploys.
  • 503 Service Unavailable: GPU pool exhausted, queue depth exceeded, or health check failing. The most frequent during traffic spikes.
  • 504 Gateway Timeout: Inference exceeded upstream timeout; large context windows are culprits.

Knowing the subtype guides mitigation. 503 suggests backoff and fallback; 504 suggests reducing max_tokens or model size.

What the data actually shows (and doesn’t)

Aggregating such probes over weeks yields a distribution, not a single number. The 5xx error rate LLM API clients see is usually low in steady state—often less than one percent—but it is bursty. Incidents are not uniform: a 30-minute window can produce 40% 503s while the surrounding hours are clean.

Public postmortems confirm this. OpenAI’s November 2023 outage generated widespread 5xx for multiple hours; Anthropic’s 2024 capacity events produced localized 503s. Without client-side measurement you are blind to the localized part.

A mistake is to compute a monthly average and call it done. If your SLO is 99.9%, a single burst of 500s for 10 minutes per week already eats your error budget if you ignore fallback. The 5xx error rate LLM API matters at percentile, not average. Track p95 and p99 error ratio per hour.

Tradeoffs: active probing vs production instrumentation

Active probing is cheap and isolated but doesn’t reflect real traffic shape. Production instrumentation captures real payloads and model diversity but mixes causes (bad prompts, provider bugs). My recommendation: run both.

  • Active probe: tiny requests, frequent, from same VPC. Good for alerting on provider-wide degradation.
  • Passive logging: tag every production call with provider, model, status, latency. Aggregate 5xx per provider per day.

Example aggregated metric document

{
  "provider": "openai",
  "model": "gpt-4o-mini",
  "window": "2024-06-01T12:00:00Z/1h",
  "total": 48210,
  "5xx": 142,
  "p95_latency_ms": 820,
  "notes": "elevated 503 after 12:40 deploy"
}

The passive approach reveals that certain models (large context windows) trigger 504 more often because of longer compute.

Designing for fallback without hiding the signal

When a provider returns 5xx, you should retry with backoff, and if persistent, route to another provider. This is where a gateway helps. n4n.ai, for instance, provides automatic fallback when a provider is rate-limited or degraded, shielding your app from transient 5xx spikes across 240+ models behind one OpenAI-compatible endpoint. But you must still meter the underlying failures; otherwise you’ll mask a chronically flaky provider behind successful fallbacks and never fix your routing weights.

Implement client-side logic like:

def call_with_fallback(request, primary, secondary):
    for provider in (primary, secondary):
        try:
            r = provider.call(request, timeout=15)
            if r.status_code >= 500:
                continue  # treat as failure, try next
            return r
        except TransportError:
            continue
    raise AllProvidersDown()

The trap: if you fallback blindly and never log the 5xx, you lose the ability to penalize a bad provider. Keep a rolling counter of 5xx per provider and shed traffic when threshold exceeded.

Alerting on the metric

Set alerts on p95 5xx ratio over 5-minute windows. Example: if provider X exceeds 5% 5xx for 10 minutes, page on-call. Use a deadman switch: if probes stop, alert too. A simple Prometheus rule:

alert: HighLLM5xxRate
expr: rate(llm_requests_total{status=~"5.."}[5m]) / rate(llm_requests_total[5m]) > 0.05
for: 10m
labels:
  severity: page

Honest tradeoffs of measuring

Measuring 5xx costs money: tokens, compute, engineering time. Over-probing can itself trigger 429s that look like degradation. Keep probe payloads minimal (max_tokens=1) and frequency modest (every 30-60s). Another tradeoff: cross-region variance means a single probe location is biased. If your users are global, run probes from at least two regions.

Also, some providers count a 503 as a billable attempt? No, they don’t bill on 5xx, but they might bill on 429? Usually not. So measurement cost is purely infrastructure.

Takeaway

The 5xx error rate LLM API surface is the clearest indicator of provider health you can get, and it is not what their status pages show. Build a small client-side probe, instrument production calls, track per-provider 5xx at p95 not average, and design fallback that preserves the failure signal. Do that, and your LLM stack stops being a hostage to provider marketing.

Tagserror-ratesreliabilityapi-errorsbenchmark

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceProvider reliability benchmark: timeout rates by modelAnalysis of LLM timeout rate benchmark across providers and models, showing why timeout variance matters and how to build resilient fallback with code.Benchmarks & performanceTracking error rates across 10 inference providersA static LLM provider error rate benchmark across 10 inference providers hides more than it reveals. Learn how to track errors continuously and route around failures.Benchmarks & performanceHow often do LLM providers hit 99.9% uptime?Analyze whether the LLM provider 99.9% uptime SLA matches real-world API reliability, with failure modes, measurement code, and fallback design.Benchmarks & performanceLLM provider reliability during peak traffic hoursAnalysis of LLM provider reliability peak hours: why single-vendor setups fail under load, how to measure degradation, and fallback patterns that hold.Benchmarks & performanceReliability benchmark for GPU-scarce open-weight modelsAnalyzes how GPU scarcity undermines open-weight model reliability and provides a benchmark methodology for measuring degradation, fallback, and tail latency.Benchmarks & performanceTracking rate-limit errors across LLM providers in 2026A practical analysis of LLM rate limit errors benchmark methodology across providers in 2026, with code for tracking and mitigation strategies.

All provider uptime and reliability benchmarks posts →