Measuring how much a cold start adds to latency

A cold start latency benchmark across models forces an uncomfortable conclusion: the seconds you lose to loading weights dominate tail latency far more than the model’s compute speed. If you treat every request as equally likely to hit a warm replica, you will misdesign your timeout budgets and retry logic.

The thesis: cold start is a fixed cost, not a scaling cost

Cold start latency is the time from request receipt to first token when no live instance holds the model weights. This cost is largely independent of prompt size and output length. Warm inference latency, by contrast, scales with the number of generated tokens and the cost per forward pass.

That means the cold start penalty is a fixed tax. For a short completion, it can be the entire latency. For a long stream, it becomes a rounding error. Engineers who only watch p50 under warm traffic get surprised by p99.

A cold start latency benchmark across models should therefore report absolute seconds to first token, not tokens per second. Throughput numbers lie when the denominator is missing the load phase.

What actually happens during a cold start

Weight loading and memory allocation

The serving process must allocate GPU memory and copy weights from CPU RAM or disk. In fp16, a 7B model needs roughly 14GB; a 70B model needs roughly 140GB. The latter does not fit on a single consumer GPU and requires tensor parallelism across multiple devices, which multiplies allocation overhead and inter-process coordination.

Memory allocation alone is not the whole story. The loader must deserialize a checkpoint format (safetensors, GGUF, or proprietary blobs) and reshape tensors into the layout the kernel expects. This is CPU-bound and benefits from fast storage, but it is never free.

Kernel and runtime warmup

CUDA kernels are lazily compiled or captured. The first attention forward pass triggers graph capture or autotuning. Many serving stacks (vLLM, TensorRT-LLM, SGLang) do a dummy forward pass on startup precisely to hide this from the first user—but if that warmup did not happen, the first request pays it.

Operators who rely on “serverless” GPU functions often skip warmup to save cost. The result is a cold start that includes both weight load and kernel init. You are not billed for the warmup, but your user is billed in latency.

How to measure it without lying to yourself

A minimal timing harness

Use a streaming call and measure time to first delta. This avoids counting generation time.

from openai import OpenAI
import time, os

client = OpenAI(
    base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
    api_key=os.environ["LLM_API_KEY"],
)

def time_to_first_token(model: str, prompt: str) -> float:
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
        max_tokens=1,
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            return time.perf_counter() - start
    return time.perf_counter() - start

Run this against a model that you know was just evicted. To force eviction, delete the deployment or wait for the provider’s idle TTL.

Controlling for cache and routing

Provider-side prompt caching can mask cold starts if the system reuses a warmed KV cache. Disable caching or use a unique prompt namespace per measurement. If your gateway supports routing directives, send a header that forces a cold replica:

client.chat.completions.create(
    model="meta-llama/llama-3-70b-instruct",
    messages=[{"role": "user", "content": prompt}],
    extra_headers={"x-prefer-warm": "false"},
)

A cold start latency benchmark across models must vary the model identifier across runs so the gateway cannot quietly reuse a warm slot.

A curl-based cross-check

A shell loop gives you a quick relative view across several model sizes:

for m in meta-llama/llama-3-8b mistralai/mistral-7b; do
  curl -s -o /dev/null -w "$m %{time_starttransfer}\n" \
    -H "Authorization: Bearer $KEY" \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"max_tokens\":1}" \
    https://your-gateway.example/v1/chat/completions
done

time_starttransfer captures the gap before the first byte, which correlates with time-to-first-token for streaming proxies.

Why most dashboards hide the problem

Most APM tools sample on successful requests. Cold starts often trigger client-side retries, so the original slow request is discarded and only the retry—which may hit a warm replica—is recorded. This biases your latency histogram toward the optimistic case.

If you instrument with a distributed trace, tag the first attempt explicitly. Otherwise your p99 will look like your warm p99 and your users will still be timing out.

Patterns from a cold start latency benchmark across models

Across public model sizes and common GPU classes, a few patterns hold without needing fabricated numbers:

• Small models punch above their weight in percentage terms. A 1B model that generates tokens in single-digit milliseconds still pays the same process-spawn and allocation tax as a 7B. The cold start can be 100x the warm first-token time.
• Large models have worse absolute but better relative penalty for long outputs. Loading 140GB takes real seconds, but if you stream thousands of tokens, the amortized cost shrinks.
• Quantized variants reduce memory bandwidth pressure but not allocation round-trips. A 4-bit 70B loads faster than fp16, yet still requires the same number of CUDA context creations.

The benchmark is only meaningful in your own account. Provider scheduling, GPU SKU, and region all shift the absolute values. A cold start latency benchmark across models run in us-east-1 on A100s tells you nothing about eu-west-2 on L40S.

Tradeoffs of eliminating cold starts

Warm pools cost money

Keeping a replica warm means renting GPU memory 24/7. For a 70B model on multi-A100, that is a dedicated cluster. Many teams shard traffic across a few warm models and accept cold starts for long-tail variants.

Fallback routing hides latency

A gateway that detects a cold or rate-limited provider and reroutes to a warm one converts a multi-second stall into a sub-second hit. This is not free—it assumes redundant capacity somewhere. An OpenAI-compatible gateway such as n4n.ai honors client routing directives and applies automatic fallback when a provider is degraded, which lets you express “avoid cold regions” without custom code.

You can also forward cache-control hints so the provider keeps weights resident:

client.chat.completions.create(
    model="mistralai/mixtral-8x7b-instruct",
    messages=[{"role": "user", "content": "repeat: hello"}],
    extra_headers={"x-cache-ttl": "600"},
)

When cold start is acceptable

Batch jobs, offline eval harnesses, and one-off summarization tasks could not care less about a 5-second preamble. The mistake is treating interactive traffic like batch traffic.

Interpreting the benchmark honestly

One sample is not a benchmark. Cold start variance is high because the underlying storage and scheduler are shared. Take 20 evicted measurements per model, drop the min and max, and report median plus p90. If you cannot evict reliably, rent a fresh namespace per run.

Do not compare a cold small model against a warm large model. The cold start latency benchmark across models only earns its name when every data point shares the same eviction state.

Takeaway: design for the warm path

Measure cold start separately from warm latency, and set your client timeout to the cold budget only for requests that cannot wait. Pre-warm the models that drive revenue; let obscure checkpoints cold-start. Use routing headers to pin critical traffic to warm replicas, and rely on fallback to absorb the rest.

A cold start latency benchmark across models is not a one-time task. Re-run it when you change regions, GPU types, or serving stacks. The fixed tax never disappears—but you can choose who pays it.