Cold starts and latency on serverless GPU platforms

Serverless GPU cold start latency agents encounter in production often turns a snappy copilot into a 20-second spinner before the first token ever appears. The promise of pay-per-second GPU compute collapses under its own economics the moment you need sub-second interactive responses in a multi-step agent loop. This analysis argues that while serverless GPU platforms are excellent for batch and sporadic inference, you must architect explicit warmness or fallback paths for latency-sensitive agents, or your users will feel every cold container pull.

Why cold starts are intrinsic to serverless GPU

Serverless means no reserved capacity. The platform scales to zero. When a request arrives for a model that isn’t resident, the scheduler allocates a GPU, pulls a container image, mounts weights, and initializes the CUDA context. For LLMs, model weights are large; a 7B model in fp16 is ~14GB, a 70B model ~140GB. Even over a fast network or local NVMe, moving that data takes seconds. Then the inference server (vLLM, TensorRT-LLM, HuggingFace TGI) builds KV cache pools, captures CUDA graphs, and warms up kernels. That sequence is the cold start.

This is not a bug. It is the cost side of the same coin that gives you zero cost when idle. You cannot wish it away with better application code; you can only change the probability of hitting it or mask the hit when it happens.

A minimal probe shows the shape of the problem:

import time, openai

client = openai.OpenAI(base_url="https://gpu.example.com/v1", api_key="x")
start = time.perf_counter()
stream = client.chat.completions.create(
    model="llama-70b",
    messages=[{"role": "user", "content": "ping"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(f"TTFT: {time.perf_counter()-start:.2f}s")
        break

If that prints TTFT: 17.84s, you just met a cold start. On the same platform, a warm call to the same model often returns first token in <300ms. The gap is purely infrastructure, not model speed.

How the agent loop amplifies cold starts

A typical agent runs a planner, calls tools, reflects, and calls the model again. Each step may target a different model or the same model on a different cold container. Serverless GPU cold start latency agents suffer because the system is chatty and stop-and-go.

Consider a realistic loop with distinct models:

def run_agent(query):
    plan = llm.generate(model="llama-70b", prompt=plan_prompt(query))
    for step in parse_plan(plan):
        emb = embed(model="embed-small", text=step.context)
        tool_out = retrieve(emb)
        draft = llm.generate(model="mixtral-8x7b", prompt=step.with(tool_out))
        if draft.is_final:
            return draft

Three model endpoints, each serverless. If the platform idles containers after 30 seconds, and your tool call takes 35 seconds, the next generate is cold again. The agent’s inherent pause-for-tool rhythm is the worst-case access pattern for scale-to-zero GPU.

The serverless GPU cold start latency agents see in multi-region deployments is even worse: each region has fewer warm replicas, so the probability of a cold hit in a secondary region spikes during failover.

Measure before you optimize

You cannot manage what you don’t instrument. Log time-to-first-token (TTFT) and total latency per agent step, tagged with a cold flag derived from a baseline you observe, not a guessed constant.

{
  "trace_id": "a1b2",
  "step": 2,
  "model": "mixtral-8x7b",
  "ttft_ms": 18200,
  "cold": true,
  "region": "us-east"
}

Aggregate p50/p95/p99 TTFT separately for first-call-in-session vs subsequent. If p99 cold is 10× p50 warm, you have a cold start problem, not a model throughput problem. Track the cold ratio: cold_calls / total_calls per model. A 5% cold ratio on a 15s penalty already ruins a 500ms p50 experience.

Mitigation patterns that actually work

Provisioned concurrency / warm pools

Most serverless GPU providers offer min-instances or provisioned concurrency. Set a minimum of 1–2 replicas for the primary agent model. You pay for idle GPU, but a single A10G at a few hundred dollars per month is cheaper than churned users. For a 70B model on A100, the math is harsher; reserve only if traffic justifies.

Route to small models for latency-critical steps

Use a 7B or 3B model for intent classification, tool selection, or reflection where a 70B adds little. Smaller weights load faster, so cold starts are shorter. In your agent code:

def select_tool(query):
    return llm.generate(model="distil-7b", prompt=query, max_tokens=16)

The large model stays reserved; the small model tolerates occasional cold starts because its wake-up time is a fraction.

Gateways with fallback and cache hints

An inference gateway can mask cold starts by honoring client routing directives and automatically falling back when a provider is rate-limited or degraded. n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models and forwards provider cache-control hints, so you can pin a system prompt to cache and shift a cold large-model call to a warm smaller one without rewriting agent logic. That is a structural fix, not a cron hack.

Provider cache-control

Forward cache_control on system prompts so the provider keeps KV cache hot across calls. This does not warm the container, but it reduces per-call compute after warmup, shrinking the tail even on warm paths.

client.chat.completions.create(
    model="llama-70b",
    messages=[
        {"role":"system","content":"You are a helpful agent.","cache_control":{"type":"ephemeral"}},
        {"role":"user","content": query}
    ]
)

Synthetic keep-alive (last resort)

A cron job that sends a tiny request every 20s can keep a container alive. It works but wastes tokens and is fragile against platform rebalancing. Prefer provisioned pools.

Tradeoffs: cost, complexity, latency

Provisioned warm pools eliminate cold starts but reintroduce the fixed cost serverless avoided. A single warm A100 at public list rates for 720 hours/month is a predictable bill regardless of traffic. If your agent serves 10 requests/day, that is wasteful. Serverless GPU cold start latency agents tolerate well is only the batch case: overnight document ingestion, eval runs, offline summarization. There, scale-to-zero is pure win.

Hybrid routing adds code complexity. You maintain fallback logic, model capability matrices, and cache policies. But for agents in production, the latency win is decisive. The alternative—hoping the platform keeps your model warm—fails the moment traffic dips.

Multi-region worsens the tradeoff: warm pools in three regions triple the fixed cost. A gateway that routes to the least-cold region based on real-time health is the only sane middle ground.

Decisive takeaway

If you are building interactive agents, treat serverless GPU as a fallback tier, not your primary path. Provision warm capacity for the models in your critical loop, use small models at the edges, and put a gateway in front that can route around cold or degraded endpoints. The platforms are not broken; they are optimized for a different access pattern than the stop-and-go nature of agentic systems. Architect for that reality and your p99 stays under control.