Cold start benchmarks: vLLM vs TGI vs TensorRT-LLM

Cold start behavior decides whether your LLM endpoint survives a traffic spike or bleeds users during autoscaling. This write-up puts vLLM vs TGI vs TensorRT-LLM cold start under the same load pattern, measuring time-to-first-token from zero containers to warmed cache. The observations come from reproducible runs on A100-80GB with 7B and 70B checkpoints, not vendor slides.

What we measured

We defined cold start as the interval between launching a serving process and returning the first token for a single request, with empty weights cache and no resident process. Warm start reuses the resident model and warmed CUDA context.

Test harness

A single Python client issued one streaming completion at process start. We killed the container between runs to force a true cold path.

import time, openai

def ttft(base_url, model):
    c = openai.Client(base_url=base_url)
    start = time.perf_counter()
    stream = c.completions.create(
        model=model, prompt="def add(a,b):", stream=True
    )
    for chunk in stream:
        if chunk.choices[0].text:
            return time.perf_counter() - start

Each server ran isolated with no weight pre-stage. We repeated five times per build and took the median.

Model set

• Mistral-7B (dense, BF16, safetensors)
• Llama-3-70B (tensor parallel 4, FP8 where supported)

We did not count Hub download time; weights were local to avoid network noise.

Capabilities

vLLM implements PagedAttention and continuous batching, and loads most Hugging Face architectures with a one-line flag. It also supports hot-loading LoRA adapters, which lets you serve many tenants without a full cold reload. TGI (Text Generation Inference) offers similar batching, built-in GPTQ/AWQ quantization, and a Rust router with native token streaming. TensorRT-LLM restricts you to NVIDIA GPUs and requires a model definition plus an engine compile, but delivers kernel-fused in-flight batching that beats the others on raw throughput for fixed shapes.

The vLLM vs TGI vs TensorRT-LLM cold start gap is rooted in these architectures. vLLM and TGI stream weights from disk into VRAM. TensorRT-LLM must either deserialize a prebuilt .engine blob or compile graphs on the fly—a step that dominates startup and is easy to get wrong.

Cost model

All three are open-source; your bill is GPU hours and engineering time. vLLM and TGI run on any CUDA GPU, so a 7B model fits on a T4 at low cost. TensorRT-LLM needs Ampere or newer and often larger host memory for engine artifacts.

The hidden cost is ops. TRT-LLM’s compile step breaks on every weight update; you maintain a build pipeline and version the engines. vLLM and TGI pull a new tag and restart. For teams rotating models weekly, that difference dwarfs the per-token savings.

Latency and throughput

Cold start penalty scales with parameter count and tensor-parallel degree. vLLM and TGI loaded Mistral-7B in roughly 10–20 seconds to first token on a single A100. Llama-3-70B across four GPUs landed in the 20–40s range. TensorRT-LLM with a prebuilt engine matched those numbers, but a from-scratch build added multiple minutes before the port accepted connections.

Warm throughput ranks TRT-LLM > vLLM ≈ TGI for fixed batch sizes. Under heterogeneous request sizes, vLLM’s scheduler wins because PagedAttention avoids padding waste.

# vLLM, tensor parallel 4
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-70B --tensor-parallel-size 4

# TGI, same model
docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3-70B --num-shard 4

# TRT-LLM build (cold node tax)
trtllm-build --checkpoint_dir ./llama3-70b-ckpt \
  --output_dir ./llama3-70b-engine --gemm_plugin fp8

The vLLM vs TGI vs TensorRT-LLM cold start table below summarizes the trade.

Ergonomics

vLLM’s OpenAI-compatible API is the path of least resistance; your existing SDK just points at a new base URL. TGI ships a similar endpoint with Prometheus metrics and a health check that reports model_loaded. TensorRT-LLM’s Python or Triton backend demands more YAML, and you write a small serving loop unless you adopt Triton.

If you front these servers with a gateway such as n4n.ai, automatic fallback masks cold start when a replica is still loading, and per-token metering keeps cost visible across model swaps. The gateway can honor client routing directives to pin a request to a warm shard.

Ecosystem

vLLM has the widest third-party integrations: LangChain, Ray Serve, KServe, and many Helm charts. TGI is native to Hugging Face spaces and the paid Inference Endpoints. TensorRT-LLM leans on NVIDIA Triton and the Dynamo stack, which is ideal if you already operate DGX clusters but a poor fit for a single EC2 box.

Limits

vLLM’s speculative decoding and quantization support lag behind TGI in some releases. TGI gates a few features (like certain routers) behind enterprise tiers. TensorRT-LLM simply will not run on non-NVIDIA hardware or unported architectures; its cold start is a non-starter for rapid model rotation because each new weight set requires a rebuild.

Head-to-head table

Which to choose

Rapid prototyping or multi-model routing

Use vLLM. Its startup is predictable and it speaks the API your code already expects. When a new checkpoint drops, you change one argument and restart. The cold start penalty is small enough that scale-to-one with a warm buffer works.

Hugging Face-centric stacks

TGI fits if you already pull from the Hub and want Rust-level routing with built-in quantization. Cold start is on par with vLLM; you trade a little peak throughput for less glue code and mature metrics.

Max throughput on frozen models

TensorRT-LLM wins only when the model is fixed and you prebuild engines in your CI image. If your deploy requires frequent weight swaps, the cold compile tax will erase the gains. Use it for high-QPS, single-model inference where every token per second counts.

Autoscaling with spiky traffic

None of these like scale-to-zero. Keep a warm pool or use a fallback gateway. For 7B class, vLLM or TGI on T4 with min=1 replicas is cheaper than TRT-LLM on A10. The vLLM vs TGI vs TensorRT-LLM cold start difference matters less than keeping one replica alive.

Pick based on how often your weights move and what hardware you already own, not on a single throughput screenshot.