n4nAI
Open console →
Benchmarks & performanceComparisonCold Start vs Warm Start Latency

First request after idle: cold start latency by provider

Compare cold start latency by provider across OpenAI, Anthropic, Groq, Together, Replicate, and Bedrock. Benchmark table, measurement code, and verdict.

n4n Team4 min read916 words

Audio narration

Coming soon — every post will get a voice note here.

Cold start latency by provider is the gap between sending your first request after a quiet period and getting the first token back, and it varies by an order of magnitude depending on who serves the model. If your app makes sporadic calls—slack bots, nightly batch jobs, internal tools—that gap is your real p99, not the warm inference speed vendors quote.

Why cold start is a separate axis from throughput

Warm inference latency measures token generation once weights are resident in GPU memory. Cold start measures everything before that: container spin-up, model weight download, CUDA context init, and KV-cache warm-up. Serverless GPU pools trade cost efficiency for that penalty. Dedicated endpoints absorb it via always-on reservations.

For a user-facing chat, a 5-second cold start is a dead session. For a background summarizer running every hour, it’s noise. Engineer your fallback logic around the former, ignore it for the latter.

Measuring it without guessing

You cannot trust provider marketing here. Write a script that sleeps longer than the provider’s idle eviction window (often 5–30 minutes), then times time_to_first_token. Use the OpenAI-compatible interface where possible to keep the harness uniform.

import time, openai, os

client = openai.OpenAI(
    base_url="https://api.together.xyz/v1",  # swap per provider
    api_key=os.environ["TOGETHER_KEY"],
)

def cold_call():
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
        messages=[{"role": "user", "content": "ping"}],
        stream=True,
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            return time.perf_counter() - start
    return time.perf_counter() - start

time.sleep(1800)  # idle longer than eviction window
print(f"cold start ttft: {cold_call():.2f}s")

Run this against each provider with the same model class. The delta between the first call after sleep and the second immediate call is your observed cold start latency by provider.

The providers head-to-head

We compare six serving options engineers actually route to: OpenAI, Anthropic, Groq, Together AI, Replicate, and AWS Bedrock. All support chat completions; differences are in idle behavior.

Capabilities

OpenAI and Anthropic expose only their own model families—no third-party weights. Groq serves a curated set (Llama, Mixtral, Gemma) on custom LPUs. Together and Replicate are open-model marketplaces: hundreds of checkpoints, including fine-tunes. Bedrock aggregates third-party models behind a unified API but restricts region/model pairings.

Price/cost model

OpenAI and Anthropic bill per token with no idle charge—you pay for what you stream. Groq uses the same per-token model, often cheaper for equivalent open weights. Together and Replicate run serverless: per-token plus a hidden cold-start cost in latency, not dollars. Bedrock is per-token with provisioned throughput options that eliminate cold start at fixed hourly cost.

Latency/throughput (cold start specifics)

This is where cold start latency by provider diverges hardest.

  • OpenAI / Anthropic: Flagship models are never evicted. Cold start is effectively zero; TTFT stays <400ms even after days idle.
  • Groq: LPU clusters keep popular models resident. Uncommon checkpoints may take 1–3s to load, but the default experience is sub-second.
  • Together AI: Serverless A100/H100 pool. Cold start for a 70B model is typically 2–8s; smaller models 1–3s. Subsequent calls warm at 100–300ms TTFT.
  • Replicate: Similar serverless GPU backing. Cold start 3–10s for large vision-language or LLM weights; cold start latency by provider here is the most variable due to shared queue depth.
  • AWS Bedrock: On-demand mode behaves like serverless (2–6s for some models). Provisioned throughput mode has zero cold start but requires commit.

An inference gateway such as n4n.ai can mitigate this by honoring routing directives and automatically falling back to a warm provider when another is degraded or cold.

Ergonomics

OpenAI’s SDK is the de facto standard; everyone emulates it. Groq and Together are drop-in OpenAI-compatible. Replicate forces a different schema (input dict, polling). Bedrock uses AWS SigV4 and a non-OpenAI shape, adding boto3 boilerplate. For quick prototyping, Together’s compatibility wins; for enterprise, Bedrock’s IAM integration matters.

Ecosystem

OpenAI has the largest tooling mesh (LangChain, Vercel AI, etc.). Anthropic has first-class Claude SDKs. Groq is newer but already in major frameworks. Together and Replicate benefit from HuggingFace model ports. Bedrock sits inside the AWS console, so CloudWatch and IAM are free.

Limits

OpenAI rate limits scale with tier, not model. Groq has per-model concurrency caps (often 1–4 streams). Together limits serverless parallelism unless you rent dedicated. Replicate caps max runtime at 60s—cold start eats into that. Bedrock region locks models: Claude in us-east-1 only, Llama in us-west-2.

Comparison table

Provider Cold start (observed) Cost model Open weights? SDK ergonomics Hard limits
OpenAI ~0s (always warm) Per token No OpenAI std Tier-based RPM
Anthropic ~0s Per token No Claude SDK Tier-based RPM
Groq <1s popular, 1–3s rare Per token Yes (curated) OpenAI-compat Low concurrency
Together 2–8s (70B) Per token + latency tax Yes (hundreds) OpenAI-compat Serverless parallelism
Replicate 3–10s Per token + latency tax Yes Custom + poll 60s timeout
Bedrock 2–6s on-demand, 0s provisioned Per token or hourly Yes (aggregated) AWS boto3 Region/model lock

Which to choose

User-facing chat with sporadic traffic
Use OpenAI or Anthropic. Their cold start latency by provider is nonexistent, and the per-token premium beats the engineering cost of masking stalls. If you must run open weights, Groq’s resident models are the only sub-second alternative.

Batch jobs every few hours
Together or Replicate are fine. A 5-second cold start on a 30-minute summarization run is irrelevant. Pick based on model availability—Together for text LLMs, Replicate for multimodal.

Regulated enterprise, AWS-native
Bedrock provisioned throughput. Pay the hourly reservation to zero out cold start and stay inside your VPC.

Multi-provider resilience
If you route across several of the above, encode cold-start expectations in your retry logic. A gateway that forwards cache-control and falls back on degradation keeps p99 flat when one provider’s GPU pool is cold.

Cold start latency by provider is not a bug; it’s a billing and architecture decision. Measure it for your actual idle pattern, then choose the row in the table that matches your tolerance.

Tagscold-startmulti-providerlatency-benchmarkinference-latency

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceCold start latency across model sizes: 7B vs 70B vs 405BCold start latency by model size compared: 7B vs 70B vs 405B across load time, cost, throughput, and ergonomics to guide model selection.Benchmarks & performanceCold start vs warm start: measuring LLM inference latencyEngineer-focused head-to-head: cold start vs warm start LLM inference latency across cost, throughput, ergonomics, ecosystem, and limits, with verdict.Benchmarks & performanceMeasuring how much a cold start adds to latencyA practical cold start latency benchmark across models shows warm pools matter more than model size for tail latency in LLM inference gateways and routers.Benchmarks & performanceContainer cold starts vs model load time: latency breakdownPractical latency breakdown of container cold start vs model load time for LLM serving, covering measurements, tradeoffs, and mitigation strategies.Benchmarks & performanceGPU autoscaling: cold start latency under traffic spikesAnalyzing autoscaling GPU cold start traffic spike behavior: tradeoffs between warm pools and scale-from-zero, with real latency patterns and code.Benchmarks & performanceKeeping models warm: latency savings vs infrastructure costAnalyzes the tradeoff to keep models warm latency vs cost for LLM serving, with break-even math, warm-pool strategies, and code to manage cold starts.

All cold start vs warm start latency posts →