Keeping models warm: latency savings vs infrastructure cost

The decision to keep models warm latency vs cost sits at the center of any production LLM serving architecture. Cold starts can inject multi-second to multi-minute delays into first-token latency, while a permanently warm GPU replica silently burns infrastructure budget whether or not requests arrive. The thesis here is simple: for most teams, a selectively warmed hot set combined with graceful cold-start fallback beats both “warm everything” and “scale to zero” extremes.

What actually happens on a cold start

A cold start for an LLM inference server is not just a process spawn. The worker must allocate CUDA contexts, copy weights from CPU RAM or disk into GPU VRAM, initialize the KV cache pools, and sometimes run graph compilation or kernel autotuning. For a 7B parameter model on a single mid-range GPU, this often takes a few seconds. For a 70B model sharded across multiple A100s, published community traces show 30–90 seconds before the first token can be generated.

Serverless inference providers hide this by scaling replicas to zero, but the latency penalty surfaces on the next request. If your p95 latency budget is 2 seconds, a cold start is unacceptable.

The economics of a warm replica

A warm replica is a GPU that is reserved and loaded, irrespective of traffic. Public cloud on-demand GPU rates typically fall between $1 and $4 per GPU-hour depending on instance type and region. A single 70B model may need one 8-GPU node, so the warm cost is continuous.

Contrast with cold-start cost, which is paid in user-perceived latency, not dollars directly. But latency has a conversion cost: if a chat UI takes 40 seconds to first token, users leave.

The break-even point can be framed as:

warm_cost_per_hour = gpu_hourly_rate * gpus_needed
requests_per_hour_where_cold_start_acceptable = warm_cost_per_hour / (value_of_saved_latency_per_request)

If a warm 8-GPU node costs $16/hr and you value each avoided 30-second cold start at $0.01 (in retained engagement), you need 1600 cold-start-avoided requests per hour to break even. Below that, you are overpaying.

Selective warming: the hot-set pattern

Most LLM traffic follows a power law. In a typical application, 80% of requests hit 2–3 models (e.g., a flagship chat model and an embedding model). The long tail of fine-tuned or specialized models sees sporadic use.

Keep the hot set warm. Let everything else cold start.

# deployment for hot-set model
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hotset-mixtral
spec:
  replicas: 1  # never scale below 1
  resources:
    limits:
      nvidia.com/gpu: 1

For the tail, use an inference gateway that can route around failures. An OpenAI-compatible gateway such as n4n.ai that provides automatic fallback when a provider is rate-limited or degraded can mask cold-start penalties by sending the request to a provider that already has the model loaded, turning a 60-second cold start into a cross-provider warm call.

Measuring your own tradeoff

You cannot optimize keep models warm latency vs cost without instrumentation. Track two metrics per model route:

Time-to-first-token (TTFT) distribution, split by cold vs warm.
GPU-hours billed to that route, including idle warm time.

A minimal warm-up keeper can be implemented in a few lines. This script pings a model every 5 minutes to prevent eviction from a warm pool:

import time
import os
from openai import OpenAI

client = OpenAI(base_url="https://your-gateway/v1", api_key=os.environ["KEY"])

def keep_warm(model: str, interval: int = 300):
    while True:
        try:
            client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": "ping"}],
                max_tokens=1,
            )
        except Exception as e:
            print(f"warm failed: {e}")
        time.sleep(interval)

if __name__ == "__main__":
    keep_warm("mistralai/mixtral-8x7b")

This costs one token per cycle but holds the model in VRAM. For a 70B model on 8 GPUs, that is cheaper than re-loading every call.

Using cache-control to shrink cold starts

Some providers support prefix caching via cache-control hints. Anthropic’s API accepts cache_control on prompt blocks, letting the provider reuse computed activations across requests. A gateway that forwards provider cache-control hints extends this benefit even when you switch backends.

{
  "model": "anthropic/claude-3-5-sonnet",
  "messages": [
    {
      "role": "user",
      "content": "System prompt that is long and static",
      "cache_control": {"type": "ephemeral"}
    }
  ]
}

Even if the model process cold-starts, the cached prefix avoids recomputation of the system prompt, cutting TTFT meaningfully.

Sizing the warm pool with autoscaling

A single warm replica handles only so much concurrency. If your hot model sees bursts, you need a min/max replica range. Use Kubernetes HorizontalPodAutoscaler with custom metrics on queue depth.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mixtral-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hotset-mixtral
  minReplicas: 1   # keep at least one warm
  maxReplicas: 4
  metrics:
  - type: External
    external:
      metric:
        name: llm_queue_length
      target:
        type: AverageValue
        averageValue: "5"

This keeps the keep models warm latency vs cost balance: baseline warm, scale on demand.

Why latency savings are non-linear

The user-perceived benefit of warming diminishes after a point. Reducing TTFT from 30s to 2s is massive. Reducing from 2s to 200ms is marginal for a chat app. Thus, spend warm budget on eliminating the cold cliff, not on shaving milliseconds off an already-warm model.

Traffic-based decision framework

Use this table to map traffic to strategy:

Request rate	Model size	Recommendation
> 1 req/s	any	Always warm, autoscale replicas
1 req/min	< 13B	Warm single replica, cheap GPU
1 req/min	70B+	Warm if GPU budget allows, else fallback routing
< 1 req/hr	any	Cold start, use gateway fallback

The keep models warm latency vs cost equation flips as model size grows. A small model is cheap to keep resident; a large model demands justification.

Honest tradeoffs

Warming everything simplifies code but destroys margin. Scaling to zero saves money but produces terrible tail latency that no amount of client-side spinners fixes. Selective warming adds operational complexity: you must classify models, manage deployments, and handle fallback.

There is also a middle ground: scheduled warming. If you know traffic peaks at 9am, pre-warm at 8:55. If your workload is bursty, a gateway with automatic fallback covers the gaps.

A note on metering

Per-token metering lets you attribute the overhead of warm ping requests to the correct cost center. If your gateway emits usage logs, tag the warm pings with a metadata field and exclude them from user billing. This keeps the keep models warm latency vs cost analysis honest.

Decisive takeaway

Keep models warm latency vs cost is not a boolean. Run a permanent warm pool only for the handful of models that exceed a request-rate threshold derived from your actual GPU bill and latency value. For everything else, accept cold starts or route through a fallback-capable gateway. Implement prefix caching wherever the provider allows, and measure TTFT per route weekly. Engineers who treat warming as a per-model financial decision, not a global setting, cut infra cost by 40–70% while keeping p95 latency inside product specs.