Reducing cold start latency with model weight caching

A cold start penalty hits the moment a model checkpoint isn’t already resident on the accelerator when a request lands. To reduce cold start latency model caching has to be a deliberate deployment tactic, not a hope that the OS page cache saves you, because the delta between a warm token and a load-from-disk token can be 10–100x for large weights. This guide gives an ordered path you can execute this week.

Measure the baseline before touching cache

You can’t tune what you don’t instrument. Capture the time-to-first-token (TTFT) for the first request after a process restart versus the fifth request on a steady process.

import time, openai

client = openai.OpenAI(base_url="http://gpu-node:8000/v1", api_key="x")
def ttft(messages):
    t0 = time.perf_counter()
    stream = client.chat.completions.create(model="mistral-7b", messages=messages, stream=True)
    for chunk in stream:
        if chunk.choices[0].delta.content:
            return time.perf_counter() - t0

print("cold", ttft([{"role":"user","content":"hi"}]))
# restart process or wait for eviction, then:
print("warm", ttft([{"role":"user","content":"hi"}]))

If the cold number is seconds and warm is hundreds of milliseconds, weight loading dominates your tail latency. That confirms the target and gives you a numeric before/after.

Pick the caching tier that matches request shape

Three tiers matter: GPU memory, host RAM, and local disk. GPU memory is the only true warm start; everything else is a cold start mitigated. To reduce cold start latency model caching on a budget, keep frequently used weights in host RAM as a serialized blob and memcpy to GPU on demand.

GPU resident: fastest, costs VRAM, limits concurrency.
Host RAM + mmap: weights stay in a process buffer or page cache; pays a copy to device.
NVMe weight cache: avoids network fetches, still pays deserialize and copy.

Tradeoff is explicit: pinning a 70B model in VRAM at ~140GB (float16) blocks other workloads. RAM tier frees the accelerator but adds a host-to-device transfer on each load.

import torch, threading
from transformers import AutoModelForCausalLM

_ram_cache = {}
_lock = threading.Lock()
def get_model(name):
    with _lock:
        if name not in _ram_cache:
            # load once into CPU RAM; per-worker .cuda() later
            _ram_cache[name] = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float16)
        return _ram_cache[name]

Prewarm explicitly, don’t wait for traffic

Relying on organic traffic to fill cache creates latency spikes for real users. Send synthetic requests at deploy time and on a cron aligned with autoscaler events.

curl -s http://gpu-node:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mistral-7b","messages":[{"role":"user","content":"warm"}],"max_tokens":1}'

For a fleet, parallelize:

for m in mistral-7b llama-13b; do
  curl -s http://gpu-node:8000/v1/chat/completions \
    -d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"w\"}],\"max_tokens\":1}" &
done
wait

Pitfall: some servers skip the forward pass when max_tokens=0. Use at least one token so the weight path actually executes.

Set eviction and size budgets

Unbounded caches OOM the host. Use an LRU keyed by last-access time and a byte budget, not item count.

from collections import OrderedDict
class WeightCache:
    def __init__(self, max_bytes):
        self.max = max_bytes; self.cur = 0; self.store = OrderedDict()
    def get(self, name, loader):
        if name in self.store:
            self.store.move_to_end(name); return self.store[name]
        w = loader(name); sz = w.nbytes
        while self.cur + sz > self.max and self.store:
            old, w2 = self.store.popitem(last=False); self.cur -= w2.nbytes
        self.store[name] = w; self.cur += sz; return w

Set TTL only if you deploy new weights frequently; otherwise version the cache key. Track bytes because a single 70B float16 weight is ~140GB, while a 7B is ~14GB.

Route to warm replicas

Once weights are cached, you must send requests to the node that holds them. Stateless round-robin defeats caching. Use session affinity or a routing directive that the gateway respects.

If you front your nodes with n4n.ai, its OpenAI-compatible endpoint honors client routing directives and forwards provider cache-control hints, so a header or field pinning the warm replica works without custom proxy code.

{
  "model": "mistral-7b",
  "messages": [{"role": "user", "content": "go"}],
  "route": {"key": "node3"}
}

Exact header names depend on your gateway; the point is to make routing cache-aware rather than random.

Validate under realistic load

A single warmup curl proves nothing. Run a staged test that forces eviction and reload to see if p99 TTFT holds.

import locust
class WeightUser(locust.HttpUser):
    @locust.task
    def ask(self):
        self.client.post("/v1/chat/completions",
            json={"model":"mistral-7b","messages":[{"role":"user","content":"x"}],"max_tokens":20})

Watch p99 TTFT. If it climbs after idle periods, your eviction TTL is too short or the prewarm cron stalled on a scaled-up pod.

Common pitfalls

Treating page cache as guaranteed: Linux reclaims under memory pressure; pin in-process or with mlock if you need certainty.
Prewarming only on deploy: autoscalers kill pods; re-run on every scale-up event.
Ignoring weight versioning: an old cached blob served after update causes silent quality regression.
Caching tiny models: the overhead of cache management exceeds the load cost for sub-1B checkpoints.

Tradeoffs you must accept

Reducing cold starts costs memory or disk. GPU pinning trades concurrency for latency. RAM tier trades host memory for faster loads. Local NVMe trades disk cost for avoiding network pulls. Measure marginal gain per dollar; once you’re under your p99 target, stop spending.

To reduce cold start latency model caching at the cluster level, combine explicit prewarm, LRU eviction, and cache-aware routing. That sequence consistently beats hoping the kernel bails you out.