GPU autoscaling: cold start latency under traffic spikes

The brutal reality of an autoscaling GPU cold start traffic spike is that your p99 latency can jump by orders of magnitude exactly when request volume triples. Most teams treat GPU autoscaling like stateless web tier scaling, but the weight of model binaries and CUDA initialization makes that analogy dangerous. This analysis breaks down where the time actually goes, why naive scale-from-zero policies fail under bursty LLM traffic, and what tradeoffs you must accept to keep tails manageable.

The anatomy of a cold start

A cold start for a GPU inference server is not a single event. It is a sequence of phases, each with its own failure modes and owners.

Node provisioning

When the autoscaler decides it needs another A100, the cloud control plane must allocate the physical card, attach network, and boot the VM. From zero, this commonly adds one to three minutes of wall-clock before your container runtime is even reachable. In constrained regions, it can be longer. No application-level tuning fixes this; it is purely infrastructure latency outside your codebase.

Container and model load

Once the node is up, the kubelet pulls your serving image (often several GB) and mounts a volume with model weights. A 70B-parameter checkpoint in FP16 is roughly 140GB. Even from a local NVMe cache, copying and mapping that into VRAM takes tens of seconds. From object storage on a cold node, expect another minute of download before the first cudaMalloc runs. This phase is dominated by I/O bandwidth, not compute.

CUDA context and kernel warmup

After the process starts, the CUDA driver must build contexts and load or JIT kernels. The first inference also triggers memory allocator growth and attention kernel autotuning. We have measured on internal clusters that the first real request after load can be 5–10x slower than a steady-state token, even with the model already resident. That is warmup, not loading, and it must be counted in your cold start budget.

Why traffic spikes expose the worst case

Bursty workloads and queueing theory

LLM traffic is not Poisson. It is spiky: a marketing blast, a cron job, or a viral thread can 10x requests in seconds. Autoscalers react on metrics windows (30–60s) and cooldown periods. Under an autoscaling GPU cold start traffic spike, the new capacity arrives long after the surge began. Requests queue behind a saturated warm pool, and because GPU compute is throughput-bound with low concurrency, queue depth explodes nonlinearly. An M/D/1 system at 90% utilization already shows sharp latency cliffs; GPU serving is worse because batching windows add coarseness.

Tail latency SLAs break

If your SLA promises 2s time-to-first-token for 95% of requests, a cold start that takes 90s to even start serving destroys that SLA for every request caught in the window. Worse, retries from clients amplify load. We have seen a simple curl retry loop turn a 30s cold start into a 5-minute outage because every retry landed on the same not-yet-ready replica. Backpressure signaling is absent in most naive OpenAI-compatible clients.

Measurement methodology

You cannot improve what you do not separate. Instrument each phase with explicit timestamps and own them separately.

import time, logging

def serve_request(model, req):
    t0 = time.monotonic()
    if not model.loaded:
        model.load()  # includes weight copy + cuda init
        logging.info("model_load_ms=%d", (time.monotonic()-t0)*1000)
    t1 = time.monotonic()
    if not model.warmed_up:
        model.generate(dummy_input)
        logging.info("warmup_ms=%d", (time.monotonic()-t1)*1000)
    return model.generate(req)

Export these as independent histograms. Do not aggregate node provisioning time into the same metric as inference latency; they have different owners (infra vs app) and different remediation paths.

A Kubernetes HorizontalPodAutoscaler config that scales on GPU utilization but ignores queue depth will systematically under-provision during spikes:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-infer
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: infer
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_util
      target:
        type: AverageValue
        averageValue: "70"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30

That stabilizationWindowSeconds delays reaction; fine for steady load, fatal for spikes. Add a custom metric for pending requests per replica and scale on that instead.

Mitigation strategies and their tradeoffs

Warm pools: cost vs latency

The simplest fix is to keep N idle GPU nodes spinning. A warm pool of size 2–4 absorbs most spikes under a minute. The tradeoff is brutal: you pay for GPUs that do nothing 90% of the time. For A100s at typical cloud rates, that is a five-figure monthly waste per idle card. Use warm pools only for the single most popular model family where spike frequency justifies the burn.

Predictive scaling

If your traffic has diurnal patterns, scale preemptively via cron or a forecaster. This converts cold starts into planned warm-ups. It fails on unpredictable spikes (news events). Combine with warm pool for residual risk. A simple cron that scales up 10 minutes before a known batch job beats any reactive autoscaler.

Request queuing and fallback

At the gateway, you can shed or reroute load instead of blocking. Honoring client routing directives and forwarding provider cache-control hints (as n4n.ai does) lets you pin sticky sessions to warm replicas and automatically fall back to a secondary provider when the primary is cold. That does not eliminate the cold start but hides it from the user behind a faster path.

{
  "route": { "prefer": "warm-replica", "fallback": "provider-b" },
  "cache_control": { "ttl": 300 }
}

Model caching and local NVMe

Never pull weights from network on a cold node if you can avoid it. Bake images with weights, or use a DaemonSet that pre-populates NVMe from a regional bucket during node bootstrap. This cuts load phase from minutes to seconds. For multi-model routers, keep a least-recently-used eviction cache on local disk and accept slightly higher load time for long-tail models.

Client-side backoff is not enough

Even with exponential backoff, a client cannot know if a 503 means “cold” or “dead”. Emit a Retry-After header from the serving layer that estimates remaining warmup time. This converts blind retries into coordinated waiting.

A decisive takeaway

Stop pretending GPU autoscaling is free. The autoscaling GPU cold start traffic spike is an unavoidable tax on bursty LLM systems, but you can choose who pays it: your wallet (warm pools) or your users (scale-from-zero). The engineering answer is tiered: keep a small warm pool for top models, use predictive scaling for known cycles, instrument phases separately, and route at the gateway to mask residual coldness. Teams that ignore phase separation will keep blaming “the network” while their p99 burns.

Treat cold start as a first-class SLO with its own dashboard, not a footnote in the deployment yaml. If you operate inference at scale, the only defensible architecture is one where a cold replica never sees a user request until it has logged a successful warmup generation.