Regional failover latency: same model, different providers

Failover latency across regions is the hidden tax of multi-provider LLM deployments. When you wire the same model behind two providers for redundancy, the time to detect a degraded primary and re-issue the request to a secondary in another region determines whether users see a blip or a timeout. This analysis breaks down where those milliseconds go and what you trade for resilience.

Why run the same model on multiple providers

Single-provider LLM dependencies fail in boring, predictable ways: per-minute token quotas, regional capacity drains, and hard outages. Running Llama-3-70B on Provider A in us-east-1 and Provider B in eu-west-1 gives you a second copy of identical weights served by a different stack.

The promise is simple. If Provider A returns 429 or drops the connection, you call Provider B. The reality is that the fallback path is not free, and the cost shows up as failover latency across regions.

{
  "model": "llama-3-70b",
  "providers": [
    {"name": "A", "base_url": "https://a.example/v1", "region": "us-east-1"},
    {"name": "B", "base_url": "https://b.example/v1", "region": "eu-west-1"}
  ]
}

Anatomy of a failover

A client-side fallback does at least five things after the primary fails:

1. Detect the failure (timeout, non-200, connection reset).
2. Select a secondary endpoint.
3. Open a connection to that endpoint (DNS, TCP, TLS).
4. Send auth and request-shaping headers.
5. Wait for the model to produce the first token.

Failover latency across regions is the sum of steps 1–5 minus any overlap. Detection can be tuned to sub-second with aggressive timeouts. Steps 3–4 dominate when the secondary is in a different continent from the client.

from openai import OpenAI, APITimeoutError, APIConnectionError

clients = [
    OpenAI(base_url="https://a.example/v1", api_key=K1, timeout=0.8),
    OpenAI(base_url="https://b.example/v1", api_key=K2, timeout=0.8),
]

def complete(prompt):
    for client in clients:
        try:
            return client.chat.completions.create(
                model="llama-3-70b",
                messages=[{"role": "user", "content": prompt}],
            )
        except (APITimeoutError, APIConnectionError):
            continue
    raise RuntimeError("all providers down")

That loop looks clean. It hides the fact that client for B may have no warm connection pool if it hasn’t been used recently.

Where the milliseconds hide

Connection setup

Cross-region round-trip times between the US and EU are typically 70–120 ms. TLS 1.3 adds one round trip on a fresh connection. If your client sits in Virginia and fails over to Frankfurt, you pay that RTT twice: once for handshake, once for request/response head.

Keep-alive mitigates this only if you pre-open the connection. A lazy fallback that constructs a new SDK client on demand pays the full tax every time.

curl -w "tcp:%{time_connect} tls:%{time_appconnect} total:%{time_total}\n" \
  -X POST https://b.example/v1/chat/completions \
  -H "Authorization: Bearer $K2" \
  -d '{"model":"llama-3-70b","messages":[{"role":"user","content":"ping"}]}'

Run that from us-east-1 and watch tls climb. From eu-west-1 it stays flat.

Auth and routing directives

API key validation is microseconds at the gateway. The bigger variable is cache state. If you send cache-control: max-age=300 to Provider A and get a cache hit, failing over to B loses that hit. The secondary re-encodes the prompt and recomputes the KV cache. That penalty is invisible in connection timing but visible in time-to-first-token.

A gateway that honors client routing directives and forwards provider cache-control hints can preserve intent, but it cannot teleport a cache across regions.

Warm vs cold model

Most providers keep popular open-weight models resident. If they don’t, weight loading for a 70B model can add seconds. That is not failover latency across regions; it is failover latency across cold starts. Know which one you are measuring.

Measuring without fooling yourself

Synthetic localhost tests lie. Run clients from the same geography as your users. Inject failure with a proxy that resets connections or returns 503 after a random delay.

# kill primary mid-flight
iptables -A OUTPUT -p tcp --dport 443 -d a.example -j REJECT --reject-with tcp-reset

Then measure p50/p95/p99 of end-to-end completion time under that fault. You will see the secondary’s connection tax appear as a p99 hump, not a p50 shift.

Tradeoffs of aggressive fallback

Output consistency

Same weights, different serving engine (vLLM vs TGI vs TensorRT-LLM) produce token probabilities that diverge after the first few tokens. If you retry on any non-200, a user may see two different answers for the same prompt across attempts. For chat, that looks like the model “changed its mind” mid-keystroke.

Cost and metering

Per-token usage metering means you pay for input tokens on every attempt. If the primary streams 20 tokens before dying, those tokens are billed. A streaming client that cancels fast limits waste, but non-streaming calls bill the full prompt twice.

Cache locality

Cross-region failover throws away regional prompt caches. For long system prompts, that cache miss can add hundreds of milliseconds of compute before the first token. Failover latency across regions therefore scales with prompt size.

Design patterns that keep p99 sane

Pre-warm connections to every provider at process start. Maintain a health-checked pool. Do not lazily build clients inside the request path.

Set a tight time-to-first-token timeout. Eight hundred milliseconds is a reasonable ceiling for interactive UX; beyond that, users perceive a stall.

Route by region affinity first. Only cross a region boundary when the local provider is unhealthy. This keeps ordinary traffic on the shortest path and reserves the cross-region tax for true emergencies.

A gateway like n4n.ai, with a single OpenAI-compatible endpoint covering 240+ models, can shift traffic internally when a provider is degraded, avoiding client-side connection setup entirely. The client sees one host; the gateway absorbs the failover latency across regions behind its own edge.

# client talks to one stable endpoint
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=GATEWAY_KEY)
# gateway applies fallback per routing policy
resp = client.chat.completions.create(
    model="llama-3-70b",
    messages=[{"role": "user", "content": prompt}],
    extra_headers={"x-failover": "cross-region-allowed"},
)

When to pay the cross-region tax

For interactive chat, treat cross-region failover as a last resort. Same-region multi-provider is strictly better: you avoid the RTT penalty and keep caches warm. If you only have one provider per region, accept that a regional outage will cost latency, not just availability.

For batch inference, failover latency across regions is irrelevant. A job that runs for minutes will absorb a 200 ms connection tax without notice.

The decisive takeaway: implement fallback at the network layer closest to the model, pre-establish connections, and measure the tax from your users’ geography. If you bolt lazy cross-region retries onto a client SDK, you trade a provider outage for a latency outage—and your p99 will tell the story.