Cold start latency llm api calls are notoriously hard to measure because most providers do not expose whether a model was resident in memory before your request arrived. If you loop a curl command every minute and call it a benchmark, you are measuring a warm pool sized by someone else’s traffic, not the cost of a cold invocation. This analysis lays out a measurement methodology that isolates the cold path and explains why the numbers you get are still only a lower bound for production.
What “cold start” means for LLM APIs
In serverless GPU inference, a cold start is the wall-clock time from request receipt to the moment the first token is generated after weights have been loaded from storage into device memory. Some providers run persistent replicas for popular models; others spin up containers on demand. The distinction matters: a 70B model on A100s may take many seconds to load, but if the provider keeps one replica warm for all tenants, you never see that cost.
Cold start is not the same as slow generation. Generation speed is tokens per second after the first token. Cold start is purely the pre-generation overhead. Confusing the two produces meaningless “latency” figures that get repeated in architecture reviews.
Why naive benchmarks lie
Engineers often write a script that sends N requests with a fixed delay and records round-trip time. Three biases creep in immediately:
- Warm contamination. If the delay is shorter than the provider’s idle eviction window, every request after the first hits a warm model.
- Shared tenancy noise. Another customer’s traffic can keep the model warm or congest the load balancer.
- Client-side jitter. DNS, TLS, and SDK overhead get folded into “latency.”
Consider this tempting but flawed snippet:
import openai, time
client = openai.OpenAI(api_key="sk-...")
for i in range(10):
t0 = time.time()
client.chat.completions.create(model="mistral-7b", messages=[{"role":"user","content":"hi"}])
print(time.time()-t0)
time.sleep(60)If the provider evicts after 300 seconds, you measured nine warm calls. If it never evicts for that model, you measured zero cold starts. The loop tells you nothing about cold start latency llm api behavior.
A measurement harness that isolates cold starts
To measure cold start latency llm api accurately, you must force a cold state and repeat enough times to characterize the distribution.
Triggering a true cold state
You cannot reliably force eviction on a shared provider. Instead, use a model or region that is documented as on-demand, or use a private endpoint where you control the autoscaler. A practical trick: pick a long-tail model that has near-zero baseline traffic. Alternatively, if the provider offers a “scale to zero” deployment, delete and recreate the deployment between trials.
Example using a realistic REST control plane (replace with your provider’s actual API):
# Tear down to force cold next time
curl -X DELETE https://api.provider.com/v1/deployments/llama-13b \
-H "Authorization: Bearer $KEY"
sleep 5
curl -X POST https://api.provider.com/v1/deployments \
-H "Authorization: Bearer $KEY" \
-d '{"model":"llama-13b","min_replicas":0}'Then immediately fire a single inference request and record TTFT. Do not batch; one cold trial per teardown.
Capturing time-to-first-token
Use streaming and timestamp the first chunk. Non-streaming round-trip includes queueing and network flush, which masks the moment compute started.
import openai, time
client = openai.OpenAI(api_key="sk-...")
def ttft(prompt):
t0 = time.perf_counter()
stream = client.chat.completions.create(
model="llama-13b",
messages=[{"role":"user","content":prompt}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
return time.perf_counter() - t0
print(ttft("Explain TCP handshake in one sentence."))Repeat 20–50 times with full teardown between each. Plot the percentiles. You will typically see a bimodal distribution: a cluster at warm latency (sub-second) if the teardown didn’t fully evict, and a tail at true cold load time.
Synthetic prompts vs production shape
A single short prompt minimizes generation time but may trigger a different code path than a 4K-token RAG context. Cold start is largely independent of input length, but some runtimes lazy-load attention kernels based on sequence length. If your production sends 8K contexts, benchmark with 8K.
Tradeoff: synthetic prompts are reproducible; production replays are realistic but noisy. I prefer to measure cold start with a fixed mid-size prompt (2K tokens) and separately measure steady-state generation speed with production traces. Mixing the two hides the signal.
Interpreting the distribution
Cold start latency llm api is not a single number. After collecting trials, filter out the warm cluster (e.g., anything under 1 second when you expected multi-second loads) and report percentiles on the remainder.
import statistics
cold_samples = [t for t in all_samples if t > 1.0] # seconds
print("p50", statistics.median(cold_samples))
print("p95", sorted(cold_samples)[int(0.95*len(cold_samples))])If you cannot force teardown, use an exponential backoff probe: send at 1s, 2s, 4s, 8s, … up to an hour, and watch for the step change in TTFT. The interval where latency jumps is your eviction window.
Control variables
Treat this like a systems benchmark:
- Same client machine and network path for all trials.
- Pin SDK version; some OpenAI-compatible clients add retry buffers.
- Record region and hardware class (T4, A10G, A100).
- Note time of day; provider GPU pools are tighter during peak.
Without these, cross-provider comparisons of cold start latency llm api are noise.
Gateway and routing effects
If you sit behind an inference gateway, the cold start number you observe client-side includes the gateway’s decision time. A gateway may retry on timeout, or fail over to a secondary provider that is warm. For example, an OpenAI-compatible gateway like n4n.ai that fronts 240+ models with automatic fallback will route around a cold or degraded provider, so your client sees a warm response while the original target would have been cold. That is a feature for uptime, but it means you must bypass routing directives and hit provider endpoints directly to measure raw cold start.
Honor provider cache-control hints if you use a gateway that forwards them; caching system prompts can turn a cold start into a warm one if the prefix is resident.
Tradeoffs of the teardown method
Deleting deployments is destructive and may incur control-plane rate limits. On shared APIs without control plane, you are reduced to statistical inference via the exponential probe described above. This is slower but non-invasive.
Another tradeoff: cold start times vary by GPU type and model size. Comparing a 7B on T4 to a 70B on A100 is apples to oranges. Normalize by reporting model, hardware, and region. Public anecdotes suggest 7B models load in low single-digit seconds on modern accelerators; 70B classes can exceed ten seconds on scale-to-zero endpoints. Your mileage depends entirely on the provider’s autoscaling policy.
Reporting format
Engineers should publish a minimal JSON blob with their numbers so others can reproduce:
{
"model": "llama-13b",
"provider": "scale-to-zero-private",
"region": "us-east-1",
"hardware": "A10G",
"trials": 30,
"cold_ttft_ms": {"p50": 4200, "p95": 8800, "p99": 11000},
"warm_ttft_ms": {"p50": 280, "p95": 410}
}Replace the values with your observed data. The structure lets you track regressions when a provider changes its warm pool strategy.
Decisive takeaway
Measure cold start latency llm api by forcing eviction, streaming TTFT, and collecting percentiles over many trials; never trust a single sleep-loop script. If you operate through a gateway with fallback, treat its latency as a separate SLA—measure providers in isolation. Cold start is a deployment property, not a model property: the same weights can be instant on one provider and twenty seconds on another based on their autoscaling policy. Design your system to assume cold starts happen, use provisioned concurrency for latency-critical paths, and benchmark the reality you control.