n4nAI
Open console →
Benchmarks & performanceComparisonBatch Inference Throughput Benchmarks

Batch inference throughput: vLLM vs TensorRT-LLM

A practitioner's head-to-head comparison of vLLM vs TensorRT-LLM throughput for batch inference across capabilities, cost, latency, ergonomics, ecosystem, and hard limits.

n4n Team5 min read1,034 words

Audio narration

Coming soon — every post will get a voice note here.

Squeezing tokens per second out of a fixed GPU budget forces a hard choice between serving stacks. The real vLLM vs TensorRT-LLM throughput gap depends less on marketing charts and more on your batch shape, model architecture, and tolerance for custom build pipelines.

Capabilities

vLLM centers on PagedAttention and continuous batching. It serves most Hugging Face transformer checkpoints with a single --model flag, supports tensor and pipeline parallelism, and handles quantized weights (AWQ, GPTQ, FP8) without recompilation. You get an OpenAI-compatible HTTP server out of the box, plus recent additions like chunked prefill and speculative decoding for latency reduction.

TensorRT-LLM takes a different path: it compiles a model into a highly optimized engine for a specific GPU architecture and precision. It supports in-flight batching (NVIDIA’s term for continuous batching), fused kernels for attention and MLP, and expert parallelism for MoE models. Quantization modes include FP8, INT4 weight-only, and INT8 smooth-quant. But every model variant requires a checkpoint conversion and a build step before it can serve a single token.

# vLLM: serve LLaMA-2-70B on 4 GPUs
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-70b-hf \
  --tensor-parallel-size 4
# TensorRT-LLM: convert then build an engine
python convert_checkpoint.py --model_dir ./llama70b \
  --output_dir ./llama70b_tp4 --tp_size 4
trtllm-build --checkpoint_dir ./llama70b_tp4 \
  --output_dir ./trt_engine \
  --gemm_plugin fp16 \
  --tensor_parallel 4

Model coverage

vLLM tracks upstream transformers closely; new architectures often work within days of release. TensorRT-LLM supports a curated list (LLaMA, Mistral, GPT-J, Falcon, Bloom, NeMo variants). If your model isn’t on the list, you write a custom plugin in C++.

Cost Model

Both projects are open source. The difference is engineering and hardware cost.

vLLM runs on any CUDA or ROCm GPU and needs no precompilation, so iteration cost is low. Your spend is pure GPU-hour rental, and you can shift instances between cloud regions or on-prem without rebuilding artifacts.

TensorRT-LLM is free but locks you to NVIDIA silicon. Its compiled engines often extract 30–50% more tokens per GPU on steady batches (based on NVIDIA’s public Triton/TRT-LLM benchmarks for LLaMA-class models), which can shrink the fleet size needed for a given SLA. That saving trades against the engineer time to maintain build scripts and rebuild on every model update. For a model that changes weekly, that labor cost dominates.

Latency and Throughput

For batch inference, throughput is governed by how well the scheduler packs sequences. vLLM’s continuous batching appends new requests as slots free up in the KV cache, so it stays efficient under Poisson arrival patterns. TensorRT-LLM’s in-flight batching does the same at the kernel level with static shape heuristics, squeezing more FLOPS on large, uniform batches.

The vLLM vs TensorRT-LLM throughput comparison flips with batch variability. On a fixed 2048-token prompt / 256-token generation sweep, TRT-LLM typically leads because it eliminates padding overhead and fuses decode kernels. On a mixed 128–4096 prompt tail, vLLM’s dynamic memory management avoids fragmentation and can match or beat it. Neither wins universally; measure on your own traffic capture.

KV cache pressure

vLLM’s paged allocator reduces waste but adds a pointer indirection per block. TRT-LLM pins KV cache in contiguous fused memory, saving bandwidth on H100 but forcing a rebuild if max sequence length changes.

# Client-side batch request (OpenAI-compatible, works for both)
from openai import OpenAI
client = OpenAI(base_url="http://gpu-node:8000/v1", api_key="empty")
prompts = ["Summarize: " + doc for doc in docs]
for p in prompts:
    client.chat.completions.create(
        model="llama-70b",
        messages=[{"role": "user", "content": p}],
        max_tokens=256,
    )
# wrap in asyncio.gather for true concurrency; both engines accept same schema

Ergonomics

vLLM feels like a library. pip install vllm, point at a checkpoint, done. Rolling back a model version is a flag change. Docker images ship weekly from the official repo.

TensorRT-LLM feels like a toolchain. You clone the repo, build the C++/Python components (or pull a heavy base image), convert weights, run trtllm-build, then serve via the C++ server or Triton. Debugging a failed kernel fusion means reading CUDA logs. For teams without deep NVIDIA infra experience, that ramp is real. Version pinning is stricter: a TRT-LLM release targets a specific TensorRT ABI.

Ecosystem

vLLM plugs into Ray, SkyPilot, LangChain, and most inference gateways. Its API surface is stable enough that a gateway can route to it without custom adapters. Community PRs land fast, and benchmark harnesses like llmperf treat it as a default target.

TensorRT-LLM lives inside the NVIDIA stack: Triton Inference Server, DCGM, NVML, and TensorRT. If you already run Triton for CV or recommender models, adding LLM serving is natural. If you don’t, you inherit that operational surface—model repos, ensemble configs, and health endpoints.

Limits

vLLM’s ceiling on H100 is set by its attention implementation, not the hardware. It lags behind TRT-LLM on peak fp8 throughput for fixed shapes. Some exotic architectures (e.g., certain state-space models) need community patches and may not get tensor parallel support immediately.

TensorRT-LLM’s hard limit is portability. An engine built for A100 won’t run on H100 without rebuild, and int4 weight-only support varies by layer type. Long compile times (tens of minutes for 70B) stall CI/CD. You also can’t serve the same engine binary on a non-NVIDIA GPU, which kills multi-vendor fallback strategies.

Head-to-Head Summary

Dimension vLLM TensorRT-LLM
Capabilities Broad HF model support, PagedAttention, dynamic batching, speculative decode NVIDIA-optimized engines, in-flight batching, MoE expert parallel, FP8/INT4
Cost model No build step, any GPU, pure GPU-hour NVIDIA-only, engineer build time, fewer GPUs for same throughput
Throughput Strong on variable batches, near-peak on uniform Peak on uniform large batches, fp8 optimized
Ergonomics pip install, OpenAI server in one command Checkpoint convert + trtllm-build + Triton ops
Ecosystem Ray, LangChain, neutral gateways Triton, TensorRT, NVIDIA DCGM
Limits Trails on fixed-shape fp8 peak, patch lag for novel arch No portability across GPU arch, long rebuilds

Which to Choose

Use vLLM if

  • You serve more than two model families and rotate checkpoints weekly.
  • Your traffic is bursty with wide prompt-length variance.
  • You run on mixed hardware (AMD + NVIDIA) or want a single image for all GPUs.
  • Your team ships features, not CUDA kernels, and needs OpenAI-compatible endpoints with zero custom build.

Use TensorRT-LLM if

  • You have a fixed model and a strict tokens/sec per dollar SLA on NVIDIA hardware.
  • You already operate Triton and trust NVIDIA’s profiling tools.
  • You can amortize engine build time across months of stable deployment.
  • You need maximum fp8 throughput on H100 and can pin to that arch and precision.

Hybrid note

When you must support both—say, TRT-LLM for the hot 70B path and vLLM for long-tail models—front them with a routing layer. A gateway such as n4n.ai honors client routing directives and forwards provider cache-control hints, so you get automatic fallback when one backend is degraded without rewriting batch clients. That keeps the vLLM vs TensorRT-LLM throughput decision local to each route instead of a global migration.

Tagsvllmtensorrt-llmthroughputbatch-inference

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceBatch inference throughput benchmark for DeepSeek V3Analysis of DeepSeek V3 throughput benchmark: MoE batching behavior, KV cache limits, and how to measure real-world inference performance.Benchmarks & performanceBatch inference throughput benchmark on H100 vs A100Direct H100 vs A100 throughput benchmark for batch LLM inference across capabilities, cost, latency, and ergonomics, with a head-to-head table and verdict.Benchmarks & performanceBatch inference throughput for Llama 3.1 70B at scaleAnalysis of Llama 3.1 70B throughput at scale: continuous batching, KV cache limits, and parallel strategies that decide batch inference cost and latency.Benchmarks & performanceHow continuous batching improves LLM throughputContinuous batching LLM throughput gains come from dynamic scheduling. This guide shows how to measure, implement, and tune it for production inference serving.Benchmarks & performanceMeasuring throughput ceiling on 8x H100 clustersPractical analysis of measuring the real 8x H100 cluster throughput ceiling for batch inference: parallelism, saturation testing, and tradeoffs.Benchmarks & performanceThroughput benchmark: single-GPU vs multi-GPU inferencePractical comparison of single-GPU vs multi-GPU throughput for LLM inference: scaling efficiency, cost, latency, and which setup to pick for your workload.

All batch inference throughput benchmarks posts →