Multi-provider routing: latency cost of redundancy

Adding a second LLM provider to your request path buys resilience against rate limits and outages, but it is not free. The multi-provider routing latency cost shows up as extra network hops, duplicated auth handshakes, and fallback detection logic that can add tens of milliseconds even when the primary is healthy. Treat redundancy as a tunable system parameter, not a boolean flag, and most of that tax disappears.

The baseline: single-provider latency

Before measuring redundancy, know what a single call costs. An OpenAI-compatible request to a provider in the same region typically pays 1–2 RTT for TLS and HTTP, then waits on time-to-first-token (TTFT). TTFT is dominated by queueing and prefill, not by your client code. A colocated routing gateway adds 1–3ms of serialization if it merely forwards bytes. That is the number to defend against when you add a backup.

Where the latency actually comes from

Naive implementations open a new TLS connection to a backup provider only after the primary times out. That mistake turns a 200ms inference call into a 200ms + 300ms cold connect + 200ms retry sequence. The multi-provider routing latency cost is dominated by connection setup and decision latency, not by the routing layer’s CPU time.

Cold TLS and DNS

A cold HTTPS request to an unfamiliar endpoint pays DNS resolution (often 5–20ms if cached, more if not), TCP handshake (1 RTT), and TLS 1.3 handshake (1 RTT). In a cross-region scenario, a single RTT can be 30–80ms. Two providers in different regions can easily add 100–200ms of dead time before the first token.

Auth and request shaping

Every provider expects different headers, body shapes, and auth tokens. If your routing layer normalizes an OpenAI-style request to provider X and then rebuilds for provider Y on fallback, you pay serialization cost and potentially an extra signature computation. This is usually sub-millisecond for JSON, but can spike if you verify signatures locally or call a secrets manager per request.

Fallback detection

The biggest hidden cost is deciding when to fail over. Waiting for a full request timeout (say 10s) before trying backup wastes time. Using an aggressive hedged request (fire to both, cancel one) doubles compute cost but caps latency. A middle ground: race a lightweight health check or use a short partial-response timeout based on TTFT SLO.

Fallback topologies

You choose one of three shapes:

• Active-passive with cold standby: Cheapest compute, worst latency on failure.
• Active-passive with warm pool: Connection kept alive, failover in single-digit ms if decision is fast.
• Active-active hedged: Every request sent to two providers, first response wins. Best latency, 2x token cost on slow paths.

The multi-provider routing latency cost scales with how much you prepay (warm pools, hedges) versus pay-on-failure (cold standby).

Measuring the overhead

You cannot optimize what you do not measure. Below is a minimal Python script using httpx to compare single-provider vs fallback routing with warm connections.

import httpx, time

URL_PRIMARY = "https://api.provider-a.com/v1/chat/completions"
URL_FALLBACK = "https://api.provider-b.com/v1/chat/completions"
HEADERS = {"Authorization": "Bearer <token>", "Content-Type": "application/json"}
PAYLOAD = {
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "ping"}],
    "max_tokens": 5,
}

def time_request(client, url):
    start = time.perf_counter()
    r = client.post(url, headers=HEADERS, json=PAYLOAD, timeout=5)
    return time.perf_counter() - start, r.status_code

with httpx.Client(http2=True) as client:
    # warm both
    time_request(client, URL_PRIMARY)
    time_request(client, URL_FALLBACK)

    t1, _ = time_request(client, URL_PRIMARY)
    t2, _ = time_request(client, URL_FALLBACK)
    print(f"primary warm: {t1*1000:.1f}ms, fallback warm: {t2*1000:.1f}ms")

The script isolates connection reuse. With warm keep-alive connections, the delta between primary and fallback is typically the extra routing decision branch plus any provider-specific overhead—often under 5ms. The multi-provider routing latency cost explodes only when connections are cold or fallback triggers late.

Cutting the tax

Keep pools warm

Maintain persistent connection pools to every provider you might route to. In Go, set MaxIdleConnsPerHost; in Python, reuse a httpx.Client or requests.Session. This alone removes 80% of the observable redundancy penalty.

Pre-resolve and pin

Cache DNS results with a short TTL and pin IPs via a custom resolver. Avoid repeated lookups on the hot path. A single stub resolver call per process start, refreshed in background, is enough.

Honor cache hints

Providers support cache-control on prompts. A gateway that forwards these hints avoids recomputing prompt prefixes on fallback. n4n.ai honors client routing directives and forwards provider cache-control hints, which means a failover to a second provider can reuse cached context instead of re-paying prefix cost.

Hedged requests with budget

If tail latency matters more than token cost, issue a secondary request after a 50ms delay only if the primary hasn’t returned first token. Cancel the slower one. This bounds the multi-provider routing latency cost to the p99 of the faster provider plus the hedge delay.

import asyncio, httpx

async def hedged(client, primary, fallback, delay=0.05):
    prim_task = asyncio.create_task(client.post(primary, headers=HEADERS, json=PAYLOAD))
    await asyncio.sleep(delay)
    fb_task = asyncio.create_task(client.post(fallback, headers=HEADERS, json=PAYLOAD))
    done, pending = await asyncio.wait({prim_task, fb_task}, return_when=asyncio.FIRST_COMPLETED)
    for t in pending:
        t.cancel()
    return done.pop().result()

This pattern adds at most one extra request when the primary is slow, not a full retry after timeout.

Explicit routing directives

Encode fallback order and thresholds in config, not in scattered try/except blocks. A JSON directive keeps the logic inspectable:

{
  "route": {
    "primary": "openai/gpt-4o",
    "fallback": ["anthropic/claude-3-5-sonnet", "meta/llama-3-70b"],
    "max_fallback_ms": 80,
    "cache_control": "honor"
  }
}

The max_fallback_ms field drives a hedged launch, not a blind retry.

Tradeoffs: when redundancy hurts

Redundancy is not free compute. If you hedge everything, you double your token spend on slow paths. For cheap, high-QPS classification calls, a 20ms added latency from a cold fallback may exceed the cost of an occasional 429. In those cases, single-provider with retry-after backoff is simpler.

Also, multi-provider routing complicates compliance and logging. You must correlate request IDs across providers and meter per-token usage accurately. A per-token usage metering layer is mandatory; without it you will overspend silently.

Another trap: provider feature drift. If your prompt uses a feature only supported by the primary (e.g., structured outputs), fallback may silently degrade quality. The latency saved is worthless if the answer is wrong.

Observability requirements

You need four numbers per route: p50/p99 TTFT for primary, same for fallback, fallback trigger rate, and wasted token rate from hedges. Export them as Prometheus histograms. A sudden rise in fallback trigger rate is an early signal of provider degradation—better than waiting for user complaints.

A decisive takeaway

The multi-provider routing latency cost is real but mostly structural, not intrinsic. Cold connections and late failover decisions cause the pain; warm pools, DNS caching, and hedged requests with cancelation contain it to single-digit milliseconds for healthy paths. If you need resilience, implement routing as a warm, directive-driven layer—not a panic retry—and measure the delta on your own network. Redundancy done lazily costs you seconds; redundancy done deliberately costs you peanuts.