Cold failover vs warm failover: latency compared

When you build multi-provider LLM infrastructure, the tradeoff between cold failover vs warm failover latency shapes your entire error budget. Cold failover initiates a fresh connection or model instance only after the primary fails; warm failover keeps standby connections or preloaded contexts ready. The difference appears in tail latency, not in your p50.

What the two modes mean

Cold failover

Cold failover is the naive retry. Your code calls provider A. On timeout or 5xx, it opens a new TCP/TLS session to provider B, sends auth, and waits for the model to load if the endpoint is serverless. For OpenAI-compatible REST APIs, that means a new HTTPSConnection, a fresh TLS handshake, and possibly a cold model start on the backend. Nothing is allocated until the moment of failure.

Warm failover

Warm failover pre-establishes the secondary path. You keep a connection pool to provider B alive with periodic health pings, reuse TLS sessions, and sometimes pre-warm the model with a dummy request. At failure time, you swap the active stream with near-zero setup cost. The standby path is already authenticated and routed.

Dimensions compared

The table below summarizes the head-to-head across the axes that matter to engineers shipping LLM features.

Dimension	Cold failover	Warm failover
Capabilities	Works with any stateless HTTP client; no advance config	Requires connection pooling, health checks, routing logic
Price/cost model	No idle cost; pay only on used requests	Idle connections or minimum provisioned instances incur cost
Latency/throughput	Adds full handshake + possible model cold start (hundreds of ms to seconds)	Sub-100ms switch typical; reuses established sessions
Ergonomics	Trivial `try/except` retry	Needs pool management, timeout budgets, circuit breakers
Ecosystem	Universal; works with every SDK	Needs gateway or custom middleware; some providers expose keep-alive
Limits	Bound by retry timeout and user patience	Bound by number of warm slots and memory/connection caps

Capabilities

Cold failover needs nothing beyond a compliant HTTP client. You can implement it in ten lines with tenacity or a manual loop. It supports any provider that speaks HTTP, including self-hosted vLLM or a serverless endpoint. The downside: an in-flight streaming response cannot be resumed; you re-issue the full prompt to the backup.

Warm failover demands more. You must maintain persistent clients, possibly with aiohttp or httpx pools, and decide policy: do you warm every backup model, or just the top two? If you use a gateway that honors client routing directives, you can delegate this. For example, n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models and can keep warm routes to multiple backends while forwarding your cache-control hints, turning warm failover into a header rather than a redesign. Warm setups can also preserve context caches if both providers support the same prefix caching scheme.

Price/cost model

Cold failover has zero standby cost. You pay per token on the provider you actually hit. If failovers are rare, this is cheapest. There is no charge for an idle socket that does not exist.

Warm failover either keeps connections open (cheap but not free if the provider charges per connection) or reserves capacity (e.g., a dedicated replica on a serverless GPU). For high-traffic systems, the reserved cost is often justified by avoided timeouts. For sporadic traffic, it’s pure overhead. Per-token usage metering still applies on the active path, but you now carry fixed overhead for the warm standby regardless of request volume.

Latency/throughput

This is where cold failover vs warm failover latency becomes concrete. A cold switch includes:

DNS resolution (if not cached)
TCP handshake (1 RTT, ~10–50ms same region)
TLS handshake (2 RTT, ~50–150ms)
Auth and request dispatch
Possible model load on serverless backend (worst case seconds)

A warm switch reuses an existing pooled connection. The only added cost is the time to detect failure and flush the standby request. In practice, warm failover keeps p99 under the primary’s baseline; cold failover pushes p99 into the seconds range under provider degradation.

Throughput suffers in cold mode because every retry consumes a new connection and may hit provider rate limits exactly when you’re failing over. Warm pools isolate that burst. The cold failover vs warm failover latency gap is environmental: same cloud region narrows it; cross-region widens it.

# Cold failover: simple sequential retry
import openai

def complete_cold(prompt):
    for base_url in ["https://api.provider-a.com/v1", "https://api.provider-b.com/v1"]:
        try:
            client = openai.OpenAI(base_url=base_url)
            return client.chat.completions.create(
                model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}]
            )
        except Exception:
            continue
    raise RuntimeError("all providers down")

# Warm failover: pre-built clients with pooled sessions
import openai, httpx

sessions = [
    openai.OpenAI(base_url="https://api.provider-a.com/v1",
                  http_client=httpx.Client()),
    openai.OpenAI(base_url="https://api.provider-b.com/v1",
                  http_client=httpx.Client()),
]

def complete_warm(prompt):
    for client in sessions:
        try:
            return client.chat.completions.create(
                model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}]
            )
        except Exception:
            continue
    raise RuntimeError("all providers down")

The second snippet pays connection setup once at startup, not per failure.

Ergonomics

Cold failover is a try/except. Any junior can read it. Warm failover requires you to tune pool sizes, handle stale connections, and implement health checks. If you get it wrong, you fail over to a dead warm slot.

A gateway that performs automatic fallback when a provider is rate-limited or degraded removes the ergonomic tax. You write one client, set a routing header, and the gateway holds the warm path. That’s the only reason warm failover is viable at scale without a dedicated platform team. Without such a layer, you are shipping a circuit breaker, a health probe, and a connection reaper alongside your feature code.

Ecosystem

Every LLM SDK supports cold failover because it’s just retries. Warm failover needs either provider-side session affinity (rare) or a middleware layer. OpenAI-compatible gateways, load balancers, and service meshes can provide it. If you’re on bare Kubernetes, you’ll write a sidecar. The broader ecosystem of model routers and inference gateways increasingly treats warm failover as a configuration flag rather than a code pattern, which is the right direction.

Limits

Cold failover is limited by your user’s timeout. If your SLA is 2s, a cold switch to a serverless model that takes 3s to load is a miss. Warm failover is limited by the number of models you can keep warm—you can’t pre-warm 240 models economically, so you prioritize. File descriptor limits on the client host and provider connection caps also bound warm pools.

Failure detection overhead

A hidden cost in both modes is detecting that the primary is dead. Cold failover often waits for a full request timeout (e.g., 30s) before switching, which is unacceptable. You must set aggressive connect timeouts (2–5s) and use hedged requests. Warm failover can run background health checks every few seconds, but those checks themselves consume tokens or requests if you ping with a real completion. Use a lightweight /v1/models call or TCP ping where possible.

Measurement methodology

Don’t trust vendor latency charts. Measure with your own prompts and regions. Use curl with --http1.1 and Timing-Allow-Origin if available, or instrument Python with time.perf_counter() around the first token. The cold failover vs warm failover latency gap is environmental: same cloud region narrows it; cross-region widens it.

# crude cold failover timing
time curl -s https://api.provider-a.com/v1/chat -d '{"model":"x"}' || \
time curl -s https://api.provider-b.com/v1/chat -d '{"model":"x"}'

Run this under load, not in isolation. A cold handshake measured on an empty network lies.

Which to choose

Batch jobs and eval pipelines → Cold failover. You care about cost and completeness, not milliseconds. A retry that adds two seconds is fine if the job runs for an hour. Keep a simple retry loop with exponential backoff.

Interactive chat and coding assistants → Warm failover. Users perceive anything over 800ms as broken. Pre-warm your top two providers and route via a gateway that meters per-token usage so you can attribute the standby cost. Set connect timeouts under 3s.

Spiky serverless traffic → Cold failover with a fast secondary that has no cold start (always-on small model). Warm pools cost too much when idle 90% of the time. Use a cheap fallback model to absorb the latency hit.

Regulated or offline environments → Warm failover with local vLLM replicas. You can’t tolerate external cold starts, and you control the hardware. Keep at least one hot standby replica per critical model.

High-compliance multi-region → Warm failover with regional gateways. Cross-region cold start is brutal; warm pools in each region absorb provider outages without traversing continents.

The cold failover vs warm failover latency decision is not about which is faster—warm wins—but about where the cost and complexity buy you p99 headroom. Pick cold for cheap resilience, warm for invisible resilience.