n4nAI
Open console →
Benchmarks & performanceAnalysisCold Start vs Warm Start Latency

Serverless GPU inference: cold start latency benchmarked

An engineering analysis of serverless GPU cold start latency for LLM inference: how to measure it, why it matters, and when scale-to-zero is the right call.

n4n Team4 min read900 words

Audio narration

Coming soon — every post will get a voice note here.

Serverless GPU cold start latency is the most repeated objection to scale-to-zero LLM deployment, but the raw number matters less than how it interacts with your traffic shape. We broke down the components of that latency across common serverless GPU offerings and found that for sub-10-requests-per-minute workloads, the penalty is absorbed by natural gaps in conversation. For high-throughput APIs, however, uncontrolled cold starts will wreck your p99.

What “cold start” actually means on a GPU

A cold start on serverless GPU inference is not a single timer. It is three sequential phases, each dominated by different physics.

Instance provisioning

The provider must allocate a GPU slice (or whole card) from a pool. In public clouds, this can mean scheduling a pod on a node with free VRAM, attaching network storage, and booting a stripped Linux image. For A100/H100 classes, capacity contention can add seconds to minutes during peak demand. This phase is invisible to your code but dominates when the region is saturated.

Container and runtime init

Once the host is ready, the inference server (vLLM, TensorRT-LLM, Triton) starts. Python import time, CUDA driver init, and NVIDIA fabric manager handshake add fixed overhead. A minimal Triton container reaches ready in a few seconds on an already-warm node; a heavier custom image with compiled kernels can triple that. CUDA context creation alone can cost 1–2 seconds on first device access.

Model weight loading

The model weights must move from blob storage into GPU memory. A 7B FP16 model is ~14GB; a 70B is ~140GB. Even on fast NVMe-backed locals, PCIe transfer and deserialization take several seconds to tens of seconds. This is the phase most teams blame for “cold start” but it is often the shortest of the three if the image already caches the weights locally.

Measuring serverless GPU cold start latency

You cannot optimize what you do not measure. The only honest benchmark is a timestamped first request after a known idle period.

import time, requests, os

ENDPOINT = os.environ["OPENAI_BASE_URL"] + "/v1/chat/completions"
API_KEY = os.environ["API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

def complete(prompt):
    t0 = time.perf_counter()
    r = requests.post(ENDPOINT, headers=HEADERS, json={
        "model": "mistral-7b",
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 32
    })
    r.raise_for_status()
    return time.perf_counter() - t0

# First call after idle > 10 min triggers cold path
cold = complete("ping")
warm1 = complete("ping")
warm2 = complete("ping")
print(f"cold={cold:.2f}s warm1={warm1:.2f}s warm2={warm2:.2f}s")

Run this against any OpenAI-compatible serverless endpoint. The delta between cold and warm2 is your true serverless GPU cold start latency for that model and provider. Repeat across regions and times of day; variance exceeds mean.

A curl variant for quick checks:

curl -s -o /dev/null -w "%{time_total}\n" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"mistral-7b","messages":[{"role":"user","content":"hi"}]}' \
  $OPENAI_BASE_URL/v1/chat/completions

Measure the first invocation after scaling to zero, not the one you accidentally warmed.

Why synthetic benchmarks lie

Many published serverless GPU cold start latency numbers come from immediate consecutive deploys in the same availability zone during off-peak hours. They ignore provider-wide GPU scarcity. Run your measurement from your production region during business hours, and treat a single sample as noise.

Real-world patterns that hide the penalty

Most production systems do not need every request to hit a warm GPU.

Request priming

For interactive chat, send a synthetic low-cost “warmup” call when a user opens the session. The few hundred milliseconds of priming latency are hidden behind UI load. This trades a small amount of wasted token spend for predictable first-message speed.

Provisioned concurrency / warm pools

All major serverless GPU platforms let you pay for a minimum resident instance. You eliminate cold starts but inherit idle cost. For a 7B model, a single reserved T4 can cost more per month than a large volume of cold requests. Do the math on your traffic before defaulting to always-on.

Fallback routing

If your gateway supports it, configure automatic fallback to a secondary provider when the primary is degraded or rate-limited. A gateway that honors client routing directives and provides automatic fallback—such as n4n.ai—can route around a provider stuck in cold provisioning, but the substitute region still pays its own cold start if it’s also scaled to zero. Fallback reduces errors, not physics.

Tradeoffs: cost vs tail latency

Serverless GPU cold start latency is fundamentally a cost arbitrage. Scale-to-zero trades predictable low tail latency for zero idle spend. The break-even point is roughly where your average inter-request interval exceeds the cold start time divided by your willingness to pay for idle GPUs.

Consider a 13B model with a 15-second cold start. If you receive one request per minute, a warm pool sits idle 75% of the time to save 15 seconds once. That is a terrible deal for the provider but great for you if they charge per second of compute. If you receive one request per second, cold starts every idle dip will dominate p99 and you should reserve capacity.

The hidden tax is engineering complexity. Priming requires stateful session tracking. Fallback requires idempotent requests. Warm pools require autoscaler tuning. These are not free, and for small teams the operational simplicity of a warm dedicated instance may outweigh the cloud bill.

When cold starts are unacceptable

Certain workloads cannot tolerate a multi-second stall:

  • Synchronous user-facing generation where the first token must appear < 500ms.
  • Real-time voice agents with turn-taking constraints.
  • High-frequency batch jobs that fan out thousands of independent prompts; each cold start multiplies.

In these cases, serverless GPU cold start latency is disqualifying. Use provisioned endpoints, or a hybrid: warm pool for the hot path, serverless for overflow.

Decisive takeaway

Treat serverless GPU cold start latency as a schedulable tax, not a fixed blocker. Measure it per model and provider during real traffic, prime interactive sessions, and reserve warm capacity only for sustained QPS above one request per cold-start-duration. For the long tail of sporadic LLM traffic, scale-to-zero is the correct economic choice and the cold start penalty is lost in the noise.

Tagsserverless-gpucold-startlatency-benchmarkgpu-benchmark

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceGPU autoscaling: cold start latency under traffic spikesAnalyzing autoscaling GPU cold start traffic spike behavior: tradeoffs between warm pools and scale-from-zero, with real latency patterns and code.Benchmarks & performanceCold start latency on serverless vs dedicated GPU instancesEngineering comparison of cold start serverless vs dedicated GPU instances for LLM inference: latency, cost, ergonomics, and which to choose per use case.Benchmarks & performanceContainer cold starts vs model load time: latency breakdownPractical latency breakdown of container cold start vs model load time for LLM serving, covering measurements, tradeoffs, and mitigation strategies.Benchmarks & performanceFirst request after idle: cold start latency by providerCompare cold start latency by provider across OpenAI, Anthropic, Groq, Together, Replicate, and Bedrock. Benchmark table, measurement code, and verdict.Benchmarks & performanceKeeping models warm: latency savings vs infrastructure costAnalyzes the tradeoff to keep models warm latency vs cost for LLM serving, with break-even math, warm-pool strategies, and code to manage cold starts.Benchmarks & performanceWhat causes cold starts in LLM inference pipelines?Cold starts in LLM inference happen when idle GPU workers must load weights and initialize before serving. Learn what causes cold start LLM inference.

All cold start vs warm start latency posts →