Provider reliability benchmark: timeout rates by model

Most teams pick a model based on benchmark scores and price, then discover in production that the real differentiator is whether the request comes back at all. Our latest LLM timeout rate benchmark measures exactly that: the fraction of calls that exceed client-side deadlines or return gateway errors across providers and model classes. The spread is wide enough that timeout behavior should influence model selection as much as accuracy.

What a timeout actually is

From an application standpoint, a timeout is any request that does not yield a usable response within the bound you set. That includes TCP connect hangs, slow LLM first-token latency that blows past your HTTP read timeout, and explicit 502/504 responses from a provider edge.

It does not include 429 rate limits, though in practice they often precede timeouts when queues back up. If a provider returns 503 because a replica is OOM, treat it as a timeout for reliability math.

Why model architecture changes the curve

Large mixture-of-experts or trillion-parameter dense models need more accelerator memory and longer scheduling queues. A provider packing those behind a shared endpoint will have higher tail latency, and thus higher timeout rates under load.

Smaller open-weight models (7B–34B) running on dedicated replicas typically return first token in <500ms. Their timeout rate is closer to network noise.

Cold starts and autoscaling

Serverless inference (common for open-weight models on some clouds) introduces cold starts. If the scaling controller is slow, your 10s client timeout fires before the worker is warm. Frontier models rarely cold-start because they are always on, but they throttle via queue depth.

Concurrency multipliers

Timeout rate is a function of concurrency, not just model. A provider scheduler using fair queuing will delay requests when you exceed its batch budget. Send 100 concurrent calls to a model with max batch size 8, and the tail sits in queue. Your 30s timeout becomes the failure boundary.

Run your LLM timeout rate benchmark at production-like QPS. A laptop script sending one request per second will report near-zero timeouts for every provider and teach you nothing.

Running your own LLM timeout rate benchmark

Don’t trust a single vendor’s status page. Instrument from the client.

Set an explicit timeout on the client. With the OpenAI Python SDK:

from openai import OpenAI, APITimeoutError

client = OpenAI(
    base_url="https://api.example-gateway.com/v1",
    api_key="sk-...",
    timeout=30.0,  # hard ceiling
    max_retries=0, # we handle fallback ourselves
)

def complete(model, messages):
    try:
        return client.chat.completions.create(
            model=model,
            messages=messages,
            temperature=0.2,
        )
    except APITimeoutError:
        return None

Record the None counts per model. Over a week you get a real LLM timeout rate benchmark for your traffic shape.

Separating network from model

A timeout in us-east-1 may be fine in eu-west-1. Tag requests with region. Aggregate by model+region.

{
  "model": "anthropic/claude-3.5-sonnet",
  "region": "us-east-1",
  "timeouts": 412,
  "total": 98431,
  "p99_latency_ms": 8200
}

Using curl for spot checks

curl -m 20 -X POST https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'

Exit code 28 from curl is a timeout. Useful for quick probes but not for SLO tracking.

Implementing fallback without making things worse

Naive retry amplifies load. If model A is timing out because its queue is full, hammering it three more times guarantees a cascade.

Use a fallback chain with jitter and a short budget.

import random, time

MODEL_CHAIN = ["openai/gpt-4o", "anthropic/claude-3.5-sonnet", "meta/llama-3.1-70b"]

def complete_with_fallback(messages, budget_ms=2500):
    deadline = time.monotonic() + budget_ms/1000
    for model in MODEL_CHAIN:
        if time.monotonic() > deadline:
            break
        try:
            client.timeout = max(0.5, deadline - time.monotonic())
            resp = client.chat.completions.create(model=model, messages=messages)
            return resp, model
        except APITimeoutError:
            time.sleep(random.uniform(0.05, 0.2))
            continue
    raise RuntimeError("all models timed out")

This caps total spend and avoids synchronized retries.

Where a gateway helps

If you route through a single OpenAI-compatible endpoint that fronts many providers, the fallback logic moves out of your code. n4n.ai does this by exposing one endpoint for 240+ models, triggering automatic fallback when a provider is rate-limited or degraded, and providing per-token usage metering so fallback cost is visible. That removes the need to hardcode model chains and keeps cache-control hints forwarded across hops.

Tradeoffs of aggressive fallback

Falling back to a smaller model changes output quality. For a code-completion feature, a 70B fallback may be fine. For a legal summarization task, it may be unacceptable.

You must define a quality floor. Some teams tag requests with fallback_allowed: false and accept higher timeout rates for those.

Also, fallback hides provider degradation. If you always silently switch, you never see that your primary model is unhealthy. Emit metrics per attempted model.

Common measurement mistakes

• Using default SDK retries. The OpenAI SDK doubles timeout per attempt by default, skewing your rate toward false successes.
• Mixing streaming and non-streaming. Streaming first-token timeout differs from full-response timeout; keep them in separate buckets.
• Ignoring 5xx as “not a timeout.” A 503 is an effective timeout for your user.
• Single-region tests. Provider edges fail independently; measure where your users are.

Timeout rates vs error rates

A provider can have 99.9% “uptime” per status page but a 5% timeout rate at p99 for a specific model because they return 200 slowly. Measure what your users experience, not what the provider reports.

The LLM timeout rate benchmark should be a procurement signal, not a footnote. If a frontier model shows 10x the timeout rate of a mid-size model at your concurrency, that is a real cost in abandoned sessions.

Decisive takeaway

Treat the LLM timeout rate benchmark as a first-class SLO input. Set per-model client timeouts at roughly 2x your observed p99, implement a bounded fallback chain, and prefer a gateway that automates provider failover while preserving cache hints and metering. Teams that ignore tail timeout behavior will ship features that work in demos and fail at 3pm on a weekday when the frontier model’s queue fills.