Tracking error rates across 10 inference providers

A one-time LLM provider error rate benchmark across 10 inference providers gives you a snapshot, not a strategy. Error modes differ sharply between throttling, hard 5xxs, and malformed completions, and each demands a different response in production. If you ship LLM features without per-class error tracking, you are flying blind on the one metric that actually breaks user trust.

The metric that actually matters: error class, not error count

Most published reliability numbers collapse every non-200 into a single “error rate.” That obscures the operational reality. A 429 from a provider means you hit a quota; a 503 means the service is down; a 200 with choices[0].message.content null means the model returned nothing usable. Treating these as equal is how teams pick a provider that looks healthy on paper and then falls over under real traffic.

If you optimize solely for a low aggregate LLM provider error rate benchmark, you might pick a provider that throttles aggressively but never crashes. For interactive chat, constant 429s are worse than a rare 500 because the user sees a hard block instead of a retry. For async batch summarization, the opposite holds—a 500 with exponential backoff is fine; a 429 with a 1-minute cooldown stalls the queue.

Track these separately:

• HTTP 429 – rate limit / quota exhaustion
• HTTP 500/502/503 – upstream infrastructure failure
• Timeout – client or network boundary
• Schema error – valid HTTP, invalid body (truncated JSON, missing fields, wrong types)
• Empty completion – 200 but no content or zero choices

A provider can have a 0.5% aggregate error rate composed entirely of 429s, while another has 0.5% composed entirely of 503s. The first is a billing problem; the second is an architecture problem.

Building a tracker for 10 providers

You don’t need a commercial panel to start. A cron job that hits each provider’s OpenAI-compatible endpoint with a trivial prompt, recording status and latency, is enough for a baseline. Use the same prompt and max_tokens to avoid provider-specific quirks. Sample at one request per minute per provider; that’s 14,400 calls a day across 10 providers, well within free tiers if you keep payloads tiny.

import openai, time, json
from collections import defaultdict

stats = defaultdict(lambda: {
    "calls": 0, "err_429": 0, "err_5xx": 0,
    "timeout": 0, "bad_schema": 0, "empty": 0
})

PROVIDERS = {
    "openai": ("https://api.openai.com/v1", "gpt-4o-mini"),
    "anthropic": ("https://api.anthropic.com/v1", "claude-3-haiku-20240307"),
    # ... 8 more entries mapping base_url to a known model
}

def probe(name, base_url, key, model):
    client = openai.OpenAI(base_url=base_url, api_key=key)
    stats[name]["calls"] += 1
    try:
        resp = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=5,
            timeout=10,
        )
        if not resp.choices or not resp.choices[0].message.content:
            stats[name]["empty"] += 1
        elif not isinstance(resp.choices[0].message.content, str):
            stats[name]["bad_schema"] += 1
    except openai.RateLimitError:
        stats[name]["err_429"] += 1
    except (openai.APIConnectionError, openai.APITimeoutError):
        stats[name]["timeout"] += 1
    except openai.APIStatusError as e:
        if e.status_code >= 500:
            stats[name]["err_5xx"] += 1
        else:
            stats[name]["bad_schema"] += 1

Persist stats to a time-series store (even a CSV rotated daily works). The goal is to produce a living LLM provider error rate benchmark that you own, not a screenshot from a vendor blog.

What continuous tracking reveals

In our own tracking across a similar set of providers, the spread between best and worst for 5xx was an order of magnitude during peak UTC afternoon hours. Throttling correlated with billing tier, not provider size—smaller API shops often gave generous 429s before hard limits, while hyperscalers returned 429 at precisely the documented RPM with zero slack.

A subtle failure mode appeared with one provider that began returning 200 with a truncated JSON array inside the content field. The OpenAI SDK didn’t throw, but downstream json.loads did. That never shows up in a naive error rate because the HTTP layer succeeded. You only catch it by validating the shape of every response against your expected schema.

Public incidents in 2024 took down multiple providers for minutes to hours. A static LLM provider error rate benchmark taken the week before tells you nothing about that Tuesday. Continuous tracking shows you the decay curve: error rate climbing from 0.1% to 3% over 20 minutes is an early warning a dashboard from last month misses.

Fallback is not a silver bullet

Automatic fallback sounds like the fix: if provider A errors, call B. But fallback multiplies cost and latency, and can mask a degraded primary. A gateway such as n4n.ai can perform automatic fallback when a provider is rate-limited or degraded, but you still need per-provider metrics to set routing weights and avoid cascading failures. If you blindly fall back from a throttled premium model to a slow open-weight model, you may trade a 429 for a 30-second timeout that hurts worse.

Client-side routing directives make this explicit:

{
  "model": "gpt-4o-mini",
  "route": {
    "fallback": ["anthropic/claude-3-haiku", "meta/llama-3-8b-instruct"]
  },
  "cache_control": { "ttl": 300 }
}

This forwards provider cache-control hints and honors your order. If you don’t track which link in the chain actually served the token, you can’t compute true reliability or cost. Your fallback chain is only as good as your telemetry into each hop.

Tradeoffs: aggregate gateway vs. own telemetry

Relying solely on a gateway’s aggregated health checks saves engineering time. The downside: you inherit their definition of “error.” If they retry on 429 before surfacing, your app sees lower throttling but higher p99 latency. Per-token usage metering (which n4n.ai and others provide) lets you attribute the cost of retries to the offending provider, turning reliability into a budget line rather than a vague risk.

Running your own probe alongside the gateway gives you an independent check. When the gateway reports 99.9% success but your probe sees 97% due to schema errors, you’ve found a blind spot in their validation. The two data sources should disagree sometimes; that disagreement is where the interesting failures hide.

How to act on the data: routing policy

Raw numbers are useless without a policy. Define thresholds per error class:

• If err_5xx > 2% over 10 minutes, drop provider from active pool.
• If err_429 > 10%, shift traffic to fallback but keep 10% canary to detect recovery.
• If bad_schema > 0.5%, alert on-call; likely a version rollout broke response shape.

Implement this in your gateway config or in a thin proxy:

def select_provider(metrics):
    live = []
    for name, m in metrics.items():
        if m["err_5xx"]/m["calls"] < 0.02 and m["bad_schema"]/m["calls"] < 0.005:
            live.append((name, m["err_429"]/m["calls"]))
    live.sort(key=lambda x: x[1])  # prefer least throttled
    return live[0][0] if live else "fallback-open-weight"

This is a crude but effective example. The point is that the LLM provider error rate benchmark you collected becomes an input to a control loop, not a quarterly slide.

Decisive takeaway

Stop citing a single LLM provider error rate benchmark as if it were an SLA. Stand up per-class error tracking across all ten providers you use, slice by model tier and time of day, and wire fallback with explicit routing weights informed by that data. The teams that ship reliable AI features are not those with the best provider on paper—they’re the ones who measured the failure modes that actually hit their users and built a system that routes around them automatically while keeping a human in the loop when the schema silently breaks.