Cold start latency by model size is the gap between sending a request and receiving the first token when the model weights aren’t already resident on a GPU. The curve from 7B to 70B to 405B parameters isn’t linear—each step up multiplies memory footprint, load time, and infrastructure complexity by far more than 10x. If you’re building latency-sensitive systems, understanding where the cold start tax hits is the difference between a snappy UX and a timeout.
What “cold start” actually measures
Cold start time is the wall-clock duration from the moment your HTTP request hits the inference server to the moment the first output token is generated, assuming no warm worker exists. It bundles several stages:
- Weights fetched from blob storage or local disk
- Deserialization and optional dequantization
- CUDA context init and kernel compilation
- KV cache allocator warmup
- Actual forward pass for the prompt
Warm start skips the first three stages entirely. The gap is dominated by I/O and memory bandwidth, not compute. That’s why cold start latency by model size tracks parameter count almost directly: more bytes to move, more devices to coordinate.
Head-to-head dimensions
Capabilities
A 7B model (Llama 3.1 8B, Qwen2.5 7B, Mistral 7B) handles classification, extraction, routing, and shallow chat competently. It will hallucinate on multi-step reasoning and lacks deep code synthesis.
A 70B model (Llama 3.1 70B, Qwen2.5 72B) clears most agentic loops, complex JSON schema adherence, and intermediate coding tasks. It’s the workhorse for production assistants that need reliability without frontier cost.
A 405B model (Llama 3.1 405B, Mixtral-large class sparse variants) approaches closed-model quality on reasoning, long-context synthesis, and nuanced instruction following. It’s overkill for templated work.
Price and cost model
Token pricing from providers scales superlinearly with size, but the bigger hidden cost is idle GPU allocation. A 405B deployment needs 8×80GB GPUs minimum for fp16; even when cold, those machines are expensive whether serving or loading.
7B runs on a single 24GB consumer card or a cheap cloud instance. 70B fits on 2×80GB A100s or 4×40GB with aggressive quantization. 405B demands multi-node orchestration. Cold starts waste money only if they happen often—if you scale to zero, you pay the load tax on every idle period.
Latency and throughput
This is where cold start latency by model size bites hardest.
- 7B: Weights ~14GB fp16. From local NVMe, load completes in 2–8s on a single GPU. Warm time-to-first-token (TTFT) is typically <100ms for short prompts.
- 70B: ~140GB fp16. Even with parallel loads across two devices, cold start lands in the 20–60s range. Warm TTFT 200–500ms.
- 405B: ~810GB fp16. Multi-GPU tensor parallel groups must all sync before first token. Cold start is 1–5 minutes in practice. Warm TTFT 500ms–2s depending on context.
Throughput (tokens/s) also diverges: 7B sustains hundreds of tokens/s per GPU; 405B drops to tens of tokens/s per user under tensor parallelism overhead.
Ergonomics
Serving 7B is trivial: llama.cpp, Ollama, or vLLM on a laptop. Quantized GGUF files make it embeddable.
70B requires real serving stacks—vLLM, TensorRT-LLM, or TGI—with careful TP size selection. You’ll touch CUDA_VISIBLE_DEVICES and pipeline parallelism.
405B is an ops project. You need orchestration (Ray, KServe), model sharding configs, and health checks that account for multi-minute boots. Rolling updates become dangerous without warm standby.
Ecosystem
7B has the richest fine-tune and distill community. 70B has solid instruct and function-calling variants. 405B has fewer derivatives but is often used as a teacher for the smaller sizes. All three are available behind OpenAI-compatible APIs on most gateways.
Limits
Context length is roughly comparable (32k–128k) across the family for modern generations, but effective context degradation is worse on smaller models. Hardware ceilings: 7B fits anywhere; 70B hits memory walls on single-GPU clouds; 405B simply cannot run on most available single instances without tensor parallelism.
Comparison table
| Dimension | 7B | 70B | 405B |
|---|---|---|---|
| fp16 weight size | ~14 GB | ~140 GB | ~810 GB |
| Cold start (typical) | 2–8 s | 20–60 s | 1–5 min |
| Warm TTFT (short prompt) | <100 ms | 200–500 ms | 500 ms–2 s |
| Min GPUs (fp16) | 1×24 GB | 2×80 GB | 8×80 GB |
| Self-host effort | Trivial | Moderate | Heavy |
| Best-fit task | Routing, extraction | Agentic prod | Frontier QA |
Measuring it yourself
Don’t trust vendor claims—benchmark your own route. Below is a minimal Python snippet using the OpenAI client to capture first-token latency. Point it at any OpenAI-compatible endpoint (including n4n.ai’s single endpoint covering 240+ models) and watch the delta between a fresh worker and a warm one.
import time, openai
client = openai.OpenAI(base_url="https://your-endpoint/v1", api_key="sk-...")
def ttft(model: str, prompt: str) -> float:
start = time.perf_counter()
stream = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
return time.perf_counter() - start
return -1
# Call twice: first hit likely cold, second warm
print("cold:", ttft("llama-3.1-8b", "ping"))
print("warm:", ttft("llama-3.1-8b", "ping"))Run this against 7B, 70B, and 405B variants on the same infrastructure to see cold start latency by model size in your own environment. The ratio will mirror the weight-size gap.
Mitigation strategies
- Keep a warm pool for 70B/405B if traffic is steady. Scale-to-zero only makes sense for 7B.
- Quantize: 4-bit AWQ on 70B cuts footprint to ~35GB, trimming cold start proportionally.
- Preload on idle: speculative warmup requests keep KV allocators hot.
- Route by task: use 7B for pre-filtering, escalate to 70B/405B only when needed. A gateway that honors client routing directives and forwards cache-control hints lets you encode this in headers.
- Fallback: if a 405B cold start breaches SLA, automatically degrade to 70B. When you route through a single OpenAI-compatible endpoint like n4n.ai that fronts 240+ models and provides automatic fallback on provider degradation, you still pay the underlying cold start tax but avoid stacked outages.
Which to choose
Latency-critical edge / high-QPS microservice Use 7B. Cold start is sub-10s and often avoided entirely by keeping one replica warm on cheap hardware. Anything larger blows your p99.
Balanced production assistant (most SaaS) Use 70B. Cold start of 30s is tolerable if you maintain a small warm fleet. Capabilities cover 90% of user intents at a fraction of 405B cost.
Frontier quality, low-QPS, batch or human-in-the-loop Use 405B. If requests are sparse, pre-warm on a schedule. Don’t put it directly in a latency-sensitive path without a standby replica.
Prototype / local dev Start with 7B locally via GGUF. Promote to 70B in staging to validate reasoning. Reserve 405B for final eval suites.
Cold start latency by model size is a physics problem, not a configuration bug. Size your workload to the smallest model that meets the accuracy bar, and treat every larger step as a deliberate infrastructure investment.