Measuring failover latency when a provider goes down

When a primary model endpoint starts returning 503s, the seconds your application spends switching to a backup determine whether users see a hiccup or a timeout. LLM provider failover latency is the sum of detection, reconnection, and retry overhead, not the ping time to a secondary region. This article argues that most teams measure the wrong thing, and that accurate numbers come only from injecting provider outages into live traffic.

What failover latency actually comprises

Most postmortems quote “failover took 30 seconds” based on the gap between first alert and recovery. That number conflates human response with system behavior. For an automated multi-provider setup, LLM provider failover latency is the wall-clock time from the moment a request to provider A becomes unviable to the moment a response from provider B starts streaming to the client.

Three components dominate:

1. Detection time – how long until the client or gateway decides A is dead.
2. Reconnection cost – DNS, TLS, and connection pool warmup for B.
3. Retry overhead – re-serializing the prompt, re-reading conversation state, and possibly re-emitting partial output to the user.

Detection is rarely instant. If you use a 5-second TCP timeout and a single retry, you add at least 5 seconds before any switch. If you rely on HTTP 429/503 signals, you might fail faster, but only if the provider actually returns those instead of hanging.

Detection time

Passive detection waits for a failed request. Active detection polls a health endpoint. The latter shifts latency to the background but introduces a blind spot: the health endpoint is often a separate process from the inference worker. I have seen a provider return 200 on /health while its GPU pool was OOM-killing every request. Your failover then triggers late, after user-visible timeouts.

Request teardown and reconnect

Opening a new TLS connection to a different provider costs 2–3 round trips plus certificate validation. If your HTTP client does not pool connections per host, you pay this on every failover. Connection reuse across providers is impossible; they have different certs and SNIs. Budget 50–200 ms for a cold TLS handshake to a major cloud region, more if you are crossing continents.

Retry and reconstitution overhead

If the original request was a streaming completion with 500 ms of output already sent, you cannot resume. You must open a fresh request to B, possibly with a truncated context if you implemented partial caching. That rebuild step is application-specific but often 5–20 ms of CPU; the real cost is the lost time-to-first-token the user already waited.

Why active health checks lie

A common pattern:

import requests

def provider_up(base_url):
    try:
        r = requests.get(f"{base_url}/healthz", timeout=0.5)
        return r.status_code == 200
    except requests.RequestException:
        return False

This returns a boolean that feeds a load balancer. The lie is twofold. First, /healthz may be served by a sidecar that never sees GPU contention. Second, even if the check is accurate, the check interval adds latency. At a 2-second poll, you average 1 second of blindness plus the check timeout. During that window, live requests still hit the dead provider.

Worse, health checks generate extra traffic that can itself trigger rate limits on the very provider you are trying to protect. I have watched a misconfigured checker send 50 req/s to a stub endpoint and get the whole API key throttled. Then the “healthy” provider fails exactly when you need it.

Streaming compounds the problem

If your application uses stream=True, the user sees tokens as they arrive. A failure at token 50 forces a choice: discard and restart from provider B (user sees a pause and then possibly divergent text), or attempt to continue from token 49 (requires provider B to accept a prefix, which most APIs do not support mid-stream). In practice you restart. That means the failover latency includes the time the user already waited plus the full regeneration of the prefix. Measure from the original request start, not from the failure point, to capture true user-perceived latency.

def stream_with_failover(prompt, providers):
    for p in providers:
        try:
            stream = client.chat.completions.create(model=p, messages=prompt, stream=True)
            for chunk in stream:
                yield chunk
            return
        except (ConnectionError, TimeoutError):
            continue
    raise AllProvidersDown()

This generator yields tokens until the first provider throws. If it already yielded some, the caller must signal the client to reset. That signaling round trip is part of your real LLM provider failover latency.

Measuring in production with fault injection

Synthetic tests from a single client in us-east-1 do not reflect your users. The only honest measurement of LLM provider failover latency comes from real traffic with controlled provider kills.

Implement a fault injection wrapper that, based on a sampled flag, forces a specific provider to appear down:

import random
from typing import Callable

class FaultyRouter:
    def __init__(self, real_call: Callable, inject_rate: float = 0.005):
        self.real_call = real_call
        self.inject_rate = inject_rate

    def call(self, provider: str, payload: dict):
        if random.random() < self.inject_rate:
            # Simulate provider outage locally
            raise ConnectionError(f"injected outage for {provider}")
        return self.real_call(provider, payload)

Run this in a small percentage of requests. Log the timestamp of the raised error and the timestamp when the fallback call returns the first byte. The delta is your real failover latency distribution.

Correlate with trace IDs:

{
  "trace_id": "a1b2c3",
  "primary_provider": "openai",
  "failed_at": "2024-05-12T10:22:31.441Z",
  "failover_to": "anthropic",
  "first_byte_at": "2024-05-12T10:22:31.902Z",
  "failover_ms": 461
}

Aggregate failover_ms as a histogram. You will typically see a bimodal distribution: a fast mode (sub-200 ms) when the failure is an immediate 503, and a slow mode (2–10 s) when the primary hangs until TCP timeout. Reporting LLM provider failover latency as a single average hides this bimodal reality.

The gateway factor

If you sit behind an inference gateway, some of the reconnection logic is outsourced. A gateway such as n4n.ai that provides automatic fallback when a provider is rate-limited or degraded will intercept the error and retry against a secondary without your application code knowing. That hides complexity, but it does not eliminate LLM provider failover latency—it moves it inside the gateway. You must measure from outside: send a request, kill the primary upstream, and observe time-to-first-token from the client’s perspective. The gateway’s routing decision and token metering add single-digit to low-double-digit milliseconds, which is negligible compared to detection, but you should confirm rather than assume.

Also note that gateways honoring client routing directives can let you pin a fallback order. If you specify x-n4n-fallback: ["provider-b","provider-c"], the gateway walks that list. Your measurement should vary the order to see if provider B’s cold TLS costs more than provider C’s slower time-to-first-token. n4n.ai forwards provider cache-control hints, so a failover request can carry the same cache key to the backup, mitigating cold cache penalty—but only if you actually set those hints on the initial call.

Tradeoffs: fast failover vs wasted spend

Aggressive failover reduces user pain but increases cost and complexity.

• Shorter timeouts catch hangs faster but risk bouncing off a provider that is merely slow under load. An 800 ms timeout on a 1,000-token prompt may trigger failover on a busy but healthy endpoint, doubling your spend for that request.
• Pre-warmed connections to backup providers eliminate TLS cost but consume file descriptors and possibly idle connection fees. For 240+ models across many providers, you cannot pre-warm all; you choose the top two.
• Speculative retries (racing two providers) cut latency to the winner but guarantee at least one wasted generation. For cheap small models that is fine; for a 70B inference call it doubles your bill.

Cold model caches matter. Provider B may be “up” but if your prompt hits a cache miss because no recent request warmed its prefix cache, time-to-first-token jumps. Failover latency should be measured with cache-control hints forwarded so you compare like-for-like.

A practical measurement methodology

1. Instrument every outbound provider call with span tags: provider, attempt, error_type.
2. Export a histogram llm_failover_duration_seconds labeled by from_provider and to_provider.
3. Run continuous fault injection at 0.5% of traffic, excluding paid production critical paths if needed.
4. Alert on p95 failover duration > your user-visible timeout budget (e.g., 2 s).
5. Weekly, kill a primary provider entirely in staging and watch the same metrics.

Example Prometheus recording rule:

- record: job:llm_failover_duration:p95
  expr: histogram_quantile(0.95, sum(rate(llm_failover_duration_seconds_bucket[5m])) by (le, from_provider, to_provider))

This gives you a defensible number to cite in architecture reviews. Pair it with the raw trace samples to show the bimodal spread.

Decisive takeaway

Stop trusting ping-based health checks and vendor status pages to size your resilience. LLM provider failover latency is a tail property of your request path, dominated by detection and reconnection, not by the speed of the backup model. Measure it by injecting faults into real traffic and tracking the delta from first failure to first byte from the alternate. Set your timeouts and circuit breakers from that data, not from intuition. If you use a gateway, measure through it. The teams that survive provider outages are those who have watched their own failover happen a thousand times in peace, not those who configured a health check and hoped.