Benchmarking TensorRT-LLM vs vLLM on an H100 cluster

Anyone sizing inference infrastructure for large models eventually runs a tensorrt-llm vs vllm h100 benchmark to decide which serving stack to standardize on. The two frameworks take opposite approaches to the same problem: extract maximum throughput from NVIDIA H100 GPUs without melting tail latency. After deploying both across multi-node H100 clusters for production traffic, the trade-offs are sharper than the marketing slides suggest.

Capabilities

TensorRT-LLM compiles a model into a highly specialized CUDA engine. It fuses kernels, applies weight quantization (FP8, INT4, AWQ), and uses paged KV-cache with in-flight batching. You pay upfront engineering cost for a runtime that is brutally efficient on fixed shapes.

vLLM pioneered paged attention and continuous batching in the open-source world. It loads checkpoints directly from HuggingFace, supports a broad model zoo, and treats batch composition as a dynamic scheduler problem rather than a compiled static graph.

Both support tensor parallelism across H100s. Both leverage FP8 on Hopper. The difference is that TensorRT-LLM’s optimizations are baked at build time; vLLM’s are discovered at runtime.

A minimal engine build for a 70B model on 8 H100s looks like this:

# TensorRT-LLM: convert + build
python convert_checkpoint.py --model_dir meta-llama/Llama-3-70b \
  --output_dir ./trt_ckpt --dtype fp8
trtllm-build --checkpoint_dir ./trt_ckpt \
  --output_dir ./trt_engine \
  --gemm_plugin fp8 \
  --max_batch_size 128 \
  --max_input_len 2048 \
  --max_seq_len 4096
trtllm-serve ./trt_engine --port 8000

The vLLM equivalent:

vllm serve meta-llama/Llama-3-70b \
  --tensor-parallel-size 8 \
  --dtype fp8 \
  --max-model-len 4096 \
  --port 8000

Cost Model

The H100 hour is priced identically whether you run TensorRT-LLM or vLLM. The cost divergence is in two places: engineering time and GPU utilization efficiency.

TensorRT-LLM’s build step can take 20–60 minutes per model per hardware config. When you change max sequence length or batch size, you rebuild. That is real ops labor. But once built, it squeezes 10–30% more tokens per watt on static workloads, which can let you serve the same QPS with fewer GPUs.

vLLM has zero build step. You swap a model weight path and restart. Its continuous batching keeps GPU occupancy high under erratic traffic, which often translates to better effective cost per token in spiky workloads even if peak throughput is lower.

There is no free lunch: if your traffic is a flat firehose, TensorRT-LLM wins on raw cost efficiency. If your traffic looks like a startup’s API—bursty, multi-tenant, model-hopping—vLLM’s agility dominates total cost of ownership.

Latency and Throughput

Under a fixed batch of 32 concurrent requests with 2K context, TensorRT-LLM typically posts the lowest time-to-first-token because the kernel graph is frozen. vLLM’s scheduler adds microseconds of overhead per step but stays within a few percent on median latency.

Tail latency is where the tensorrt-llm vs vllm h100 benchmark gets interesting. vLLM’s continuous batching absorbs a sudden request surge by appending to the running batch. TensorRT-LLM’s in-flight batching does this too, but only within the compiled max_batch_size and sequence limits. Blow past those and you queue at the application layer.

On decode-bound workloads, FP8 on H100 gives both frameworks roughly 1.8–2.1x the token throughput of FP16. The gap between them narrows at FP8 because memory bandwidth—not compute—is the bottleneck, and both use the same paged KV-cache strategy.

Ergonomics

TensorRT-LLM is a C++/CUDA citizen with Python bindings. You live in Docker images tagged to specific TensorRT and CUDA versions. A minor driver bump can break your engine. Debugging requires nsys profiles and reading generated kernel names.

vLLM is a pip install away. Its logs are Pythonic. Its config surface is a single CLI or a EngineArgs object. For most teams, an engineer can go from zero to a serving endpoint in an afternoon.

Client side, both expose an OpenAI-compatible REST API. A standard client works unchanged:

from openai import OpenAI
client = OpenAI(base_url="http://gpu-node:8000/v1", api_key="none")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3-70b",
    messages=[{"role": "user", "content": "Summarize this log"}],
    max_tokens=256,
)
print(resp.usage.completion_tokens)

Ecosystem

TensorRT-LLM slots into the NVIDIA stack: Triton Inference Server, NVCF, DeepStream. If you already run Triton for vision models, adding an LLM backend is natural. NVIDIA validates specific model architectures; unsupported layers mean you write a plugin.

vLLM’s ecosystem is broader in the Python ML world. Ray Serve, SkyPilot, LangChain, LlamaIndex, and OpenLLM all have first-class vLLM integrations. New model releases (Mistral, Qwen, DeepSeek) usually get vLLM support within days. TensorRT-LLM support lags by weeks unless NVIDIA prioritizes it.

Limits and Failure Modes

TensorRT-LLM fails closed when the model graph changes. Add a new attention variant and the build errors out. Its engine is hardware-locked: an engine built for H100 will not run on A100. You maintain a matrix of artifacts.

vLLM’s weakness is memory fragmentation under heterogeneous sequence lengths. With hundreds of tenants sending 100-token and 32K-token requests, the KV-cache allocator can throttle. Recent versions mitigate with enable_prefix_caching and chunked context, but you still tune --block-size and --gpu-memory-utilization by hand.

Both frameworks crash if you exceed H100 HBM with a bad batch config. vLLM fails loudly with OOM; TensorRT-LLM can silently truncate if you misconfigure max_seq_len.

Head-to-Head Comparison

Which to Choose

Choose TensorRT-LLM if: You serve a fixed set of models at massive scale, your traffic pattern is predictable, and you have CUDA engineers on staff. Examples: a consumer chatbot with one 70B model, a translation service with SLA-bound p99 latency. The upfront build cost is amortized over millions of requests.

Choose vLLM if: You are iterating on model choices weekly, serving many smaller models behind one cluster, or need to absorb unpredictable traffic. Examples: an internal LLM platform for 50 teams, a startup testing fine-tunes daily. The absence of a build step and broad compatibility outweighs the throughput edge.

Run both behind a router if: You need maximum efficiency for steady premium traffic and elastic fallback for everything else. A gateway such as n4n.ai can honor client routing directives to pin latency-sensitive calls to a TensorRT-LLM pool while using vLLM as elastic capacity, and it will automatically shift load when a provider is rate-limited or degraded. That hybrid topology is what we run in production: TRT-LLM for the 80% steady baseline, vLLM for the long tail.

The tensorrt-llm vs vllm h100 benchmark is not a trophy competition. It is a capacity planning exercise. Measure your own traffic shape before committing—the framework that wins on a synthetic 4K-context benchmark may lose on your real 300-token API calls.