What causes cold starts in LLM inference pipelines?

A cold start in an LLM inference pipeline is the latency penalty incurred when a request reaches a compute worker that has no model weights resident in GPU memory and no initialized execution context. It spans the time to allocate devices, load checkpoint tensors from blob storage, shard across accelerators, and warm up CUDA kernels before the first token can be generated. To understand what causes cold start LLM inference, you have to follow the path from idle scale-to-zero infrastructure to a ready-to-serve transformer.

What a cold start actually is

Warm vs cold execution

In a warm start, the model is already loaded, the CUDA context exists, and the KV cache pools are allocated. The request flows straight into prefill. In a cold start, none of that exists. The scheduler must bring up a worker process, possibly a whole container, attach GPUs, and run initialization.

The penalty is dominated by weight loading and kernel autotuning, not by Python import time. For a 70B parameter model in fp16, that’s 140 GB of weights. Even from a local NVMe tier, moving and reshaping that data takes seconds. The gap between warm and cold can be 20–100x in tail latency.

How LLM inference pipelines boot

The worker lifecycle

A typical serverless GPU worker follows these stages:

1. Provision: Hypervisor or container runtime allocates a GPU slice.
2. Image pull: If not cached, the inference server image (often 10–20 GB with CUDA libs) downloads.
3. Process start: The server binds ports, loads tokenizer, initializes framework.
4. Weight load: Checkpoint shards fetched from object storage, deserialized, transferred to device.
5. Compile/optimize: Forward pass graph captured (e.g., CUDA graphs, TensorRT engine build).
6. Ready: Health endpoint flips, request queue opens.

Each stage fails independently. A slow S3 bucket or a congested PCIe bus stretches step 4.

Scale-to-zero economics

Platforms default to scaling inactive model replicas to zero to save cost. That decision is the root of what causes cold start LLM inference in most managed offerings. The trade-off is explicit: pay for idle GPUs or pay latency on first touch.

# Typical curl to an OpenAI-compatible endpoint, timing the full request
curl -w "total=%{time_total}\n" -s https://api.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mistral-7b","messages":[{"role":"user","content":"hi"}]}'

If the replica is cold, time_total might show 8–15s; warm calls often finish in <1s.

Configuration knobs that affect boot

Inference servers expose flags that change cold-start behavior. For vLLM, load_format and tensor_parallel_size directly impact weight movement:

{
  "model": "meta-llama/Llama-3-8B",
  "tensor_parallel_size": 2,
  "gpu_memory_utilization": 0.9,
  "enforce_eager": false,
  "load_format": "safetensors"
}

Using safetensors avoids pickling overhead. Setting enforce_eager=true skips CUDA graph capture, trading some warm throughput for faster initial ready state.

Why it matters in production

Tail latency and UX

Users judge chat apps by time-to-first-token. A cold start injects a multi-second blank screen. For synchronous APIs, it can blow past client timeouts.

Cost amplification

Cold starts waste GPU cycles on loading instead of serving. If traffic is spiky, you repeatedly pay the load tax. Autoscalers that are too aggressive on scale-down create an oscillation: cold, warm, scale-to-zero, cold.

Routing complexity

Some gateways attempt to hide this. A gateway with automatic fallback when a provider is rate-limited or degraded can pick a different warm backend; if all are cold, you still wait. The fallback logic cannot conjure weights that were never loaded.

SLO enforcement

If your SLO is p99 < 2s, a single cold start per hour violates it. Engineers often mistakenly attribute the spike to network or prompt length. Proper instrumentation separates boot stages.

Concrete example: tracing a cold request

Assume a Python service calling a gateway. We instrument with time:

import time, openai

client = openai.OpenAI(base_url="https://api.example.com/v1", api_key="x")

start = time.perf_counter()
resp = client.chat.completions.create(
    model="llama-3-8b",
    messages=[{"role": "user", "content": "Explain cold starts."}],
)
elapsed = time.perf_counter() - start
print(f"wall={elapsed:.2f}s")

First invocation after idle:

wall=12.41s

Subsequent:

wall=0.84s

The 12s is not network; it’s the worker boot. A span breakdown from the server logs might show:

{
  "stages": {
    "container_start": 1.2,
    "weight_load": 9.8,
    "kernel_capture": 1.1,
    "prefill": 0.3
  }
}

That weight_load number is the answer to what causes cold start LLM inference in this case: pulling 16 GB of fp16 weights from a shared filesystem.

Multi-region wrinkle

If your routing sends the first request to a region that hasn’t seen traffic, you pay cold there even if another region is warm. Client-side sticky routing helps but complicates failover.

Common misconceptions

“Only serverless has cold starts”

False. A dedicated A100 with a long-idle model can evict weights under memory pressure or after a crash. Any architecture that doesn’t pin the model resident pays sometimes.

“Quantization makes cold starts disappear”

INT4 cuts weight size 4x, so load time drops. But you still must allocate, deserialize, and dequantize or run with quantized kernels. The stage exists; it’s smaller.

“Batching hides the cost”

Continuous batching amortizes compute across requests, but the first request in a cold worker triggers the load before any batch exists. Subsequent requests benefit, yet the pioneer pays full freight.

“Provider cache-control headers fix it”

Forwarding cache-control hints to providers helps reuse compiled graphs or KV caches for repeated prompts. It does nothing for the initial weight materialization. n4n.ai honors client routing directives and forwards provider cache-control hints, which can keep a warm path warm, but the first cold load is untouched.

“Small models are immune”

A 1B model still needs process start and CUDA init. Absolute seconds are lower, but relative latency vs warm can be 10–20x, same shape.

“GPU is the bottleneck”

Often the network or object store IOPS limits weight transfer, not GPU memory bandwidth. Measure before assuming.

How to reason about causes systematically

Map your pipeline into the stages above and measure each. If weight_load dominates, invest in local NVMe or weight caching. If container_start is high, use snapshotting or smaller images. If kernel_capture hurts, precompile at build time.

What causes cold start LLM inference in your stack is rarely mysterious once you trace it—it’s the absence of a ready execution context, funded by scale-to-zero or disrupted by eviction.

Mitigation that actually works

• Provisioned concurrency: Keep N replicas warm for critical models.
• Lazy loading with fast local cache: Mount a node-local SSD mirror of checkpoints.
• Model sharding precomputed: Store already-sharded tensor files to skip reshape.
• Health-check gating: Don’t route traffic until ready true, avoiding partial-cold.
• Smaller variant for spikes: Route low-priority traffic to a 3B model that cold-starts in <2s.

These don’t eliminate the physics; they move the cost off the request path.

Measurement methodology

Add spans to your inference server. If using OpenTelemetry:

from opentelemetry import trace
tracer = trace.get_tracer("inference")

with tracer.start_as_current_span("weight_load"):
    load_weights()

Aggregate p95 per stage. You will see weight_load variance dwarf other stages for cold paths.

Summary

Cold start is the tax for not keeping a transformer resident. The causes are concrete: provisioning, image start, weight transfer, kernel init. Understanding what causes cold start LLM inference lets you decide consciously between cost and latency, and instrument the exact stage that bites you.