Cold start vs warm start: measuring LLM inference latency

Measuring cold start vs warm start LLM inference latency is the difference between a user staring at a spinner for 20 seconds and getting a streamed response in under a second. The gap is driven by weight loading, CUDA context initialization, and KV cache warm-up, not by the model’s raw forward pass. This post puts the two deployment modes head to head so you can pick the right one for your traffic shape.

What the two modes actually do

Cold start path

A cold start begins with no process holding the model. The inference worker boots, pulls weights from disk or object storage, allocates VRAM, builds CUDA graphs, and compiles any JIT kernels. On a 7B model, weight loading from local NVMe can take 2–5 seconds; from a network block store it can be 10–20 seconds. Then the runtime initializes the scheduler, registers the model with the API server, and only then marks itself ready. If you use TensorRT-LLM, engine building can add minutes on first boot.

Warm start path

A warm start hits an already-running server. The model tensors sit in GPU memory, the scheduler is initialized, and often a prefix cache already holds your system prompt. The first token emits after the normal prefill step, typically 50–500 ms depending on prompt length and batch size. The process may have been idle for milliseconds or hours, but as long as it was not evicted, the cost is constant.

Below is a minimal Python probe that distinguishes them by measuring time to first byte from a local vLLM instance versus a freshly spawned server.

import time, requests, subprocess

def ttft(url, prompt="Explain cold start in one sentence."):
    t0 = time.perf_counter()
    r = requests.post(url, json={"prompt": prompt, "max_tokens": 1}, stream=True)
    for chunk in r.iter_content():
        if chunk:
            return time.perf_counter() - t0
    return None

# warm: assume server already up on :8000
print("warm ttft", ttft("http://localhost:8000/generate"))

# cold: launch server, wait, probe, kill
proc = subprocess.Popen(
    ["python", "-m", "vllm.entrypoints.api_server", "--model", "meta-llama/Llama-2-7b"]
)
time.sleep(20)  # naive wait for load
print("cold ttft", ttft("http://localhost:8000/generate"))
proc.terminate()

The naive sleep is exactly the kind of fragility you deal with in cold-start environments. In production you poll a /health endpoint that returns 200 only after the model is loaded.

Capabilities

Cold start lets you scale to zero and run thousands of distinct model variants without reserving GPUs for each. You can serve a long-tail of fine-tunes, snapshot revisions, or quantized variants on demand. The trade-off is that each variant pays the boot tax on first use.

Warm start locks a GPU to a specific model (or a small pool behind one endpoint). You trade flexibility for predictability. Some gateways expose warm endpoints with fixed model revisions; if you need to swap weights, you redeploy and reheat. Warm servers can also support features that require persistent state, like session-aware KV cache reuse across requests.

Price and cost model

Cold start is usually billed per invocation plus compute duration (GB-seconds of GPU). If you get ten requests per hour, you pay for ten boots. At scale, the repeated loading wastes GPU-seconds and inflates cost because the same weights are read from storage dozens of times per minute under burst.

Warm start is billed for reserved capacity—by the hour for a dedicated instance, or via a minimum token commitment on a managed tier. Per-token price may be lower because the provider amortizes the fixed cost. A gateway such as n4n.ai meters per-token usage and can automatically fall back when a provider is degraded, but the underlying economic split between cold and warm still holds: you either pay for idle VRAM or you pay for boots.

Latency and throughput

This is where cold start vs warm start LLM inference latency diverges hardest. Cold start adds a fixed tax of seconds before the first token. Warm start keeps tail latency bounded by the scheduler.

Throughput tells the same story. A warm server batches 32 requests with a modest prefill penalty thanks to continuous batching. A cold server handles one request, then must keep the process alive to get any batching benefit; if it scales to zero after idle, the next request pays the tax again.

{
  "cold_start": {
    "time_to_first_token_p50_ms": 8000,
    "time_to_first_token_p99_ms": 25000,
    "max_concurrent_requests_before_load": 0
  },
  "warm_start": {
    "time_to_first_token_p50_ms": 220,
    "time_to_first_token_p99_ms": 900,
    "max_concurrent_requests": 64
  }
}

Numbers above are illustrative of typical 7B–13B models on A100; your mileage varies with model size, storage class, and whether the runtime uses prefix caching. The p99 gap is what kills interactive products.

Ergonomics

Cold start forces you to handle readiness. You need health checks that distinguish “process up” from “model loaded”, request queues that buffer while booting, and client timeouts longer than the boot window. Many serverless GPU platforms expose a “warmup” hook or provisioned concurrency; use it or your first users will time out.

Warm start is just an HTTP call. You still need retry logic for provider errors, but you skip the boot race. If you run your own vLLM or TGI, warm start means writing a systemd unit, setting --max-num-seqs, and watching OOM kills. The operational surface is smaller but always on.

Ecosystem

Cold start is the default on serverless GPU offerings (Modal, Banana, Baseten scaled-to-zero, some SageMaker serverless configurations). It pairs well with event queues and async workers that can tolerate delay.

Warm start dominates dedicated inference clusters, provisioned throughput tiers, and self-hosted fleets. OpenRouter-class gateways often present a unified warm endpoint that abstracts multiple providers; n4n.ai is one such gateway with an OpenAI-compatible route to 240+ models and honors client routing directives and provider cache-control hints. You point your client at one URL and get warm routing across vendors.

Limits

Cold start breaks down under bursty traffic. If 100 requests arrive simultaneously, you spawn 100 workers or queue. Either way, the first token lags. Also, some models exceed max boot time limits of serverless platforms (commonly 60–300s). If your engine build exceeds that, cold start cannot serve you.

Warm start limits you by VRAM and batch size. A single A100 80GB warms one 70B model at int4; if you need three models hot, you need three GPUs or careful multiplexing with something like Hugging Face Text Generation Inference’s model sharding. Context length also bites: a 32k context warm server reserves KV cache memory that reduces concurrent capacity.

Head-to-head summary

Which to choose

Sporadic, low-volume, many models. Cold start wins. If you serve a long-tail of fine-tunes with <1 QPS aggregate, paying for always-on GPUs is pure waste. Accept the latency hit or hide it behind an async job queue where a 15-second delay is acceptable.

Interactive chat or agent loops. Warm start is non-negotiable. A 10-second cold start per tool call destroys UX. Provision a warm endpoint for the primary model and use cold start only for rare fallback models that trigger once a day.

Batch processing. Cold start is fine if the batch itself is large enough to amortize boot. Kick off a worker, load once, process 10k rows, then exit. The boot tax becomes noise when spread across a million tokens.

Cost-sensitive at scale. Model the math. If your warm cost per token is 30% lower and you have steady traffic, warm pays for the reserved GPU within days. Cold start’s boot tax becomes a throughput ceiling that forces you to over-provision concurrency.

Multi-provider redundancy. Use a gateway that supports both. Route interactive traffic to warm endpoints; on provider degradation, fall back to a cold start backup. The cold path absorbs the rare miss; the warm path keeps p99 sane.

The cold start vs warm start LLM inference latency trade-off is fundamentally about who pays the loading cost: the user’s request or your idle budget. Measure your own TTFT with the snippet above against your real model and storage backend before trusting any vendor’s marketing chart.