The batch size vllm vs tgi latency picture is not a fixed ranking you can paste into an architecture doc. It inverts as soon as you move from single-stream inference to saturated GPU serving. Both frameworks now support continuous batching, but their memory management and scheduling overheads respond differently to increasing concurrent sequences, which directly changes tail latency and throughput.
Architectural levers that decide latency
Understanding the batch size vllm vs tgi latency shift starts with how each engine lays out the KV cache and decides which tokens to compute next.
PagedAttention vs contiguous KV caches
vLLM allocates KV cache in fixed-size blocks, akin to virtual memory pages. This eliminates the need to reserve contiguous GPU memory per sequence and slashes fragmentation. TGI stores KV caches in contiguous tensors sized to the max sequence length, then packs them via Flash Attention’s varlen interface. At batch size 1 the contiguous approach has less bookkeeping: no page table walks, no block allocation metadata. As batch size grows, however, TGI must either pad or waste slots, while vLLM packs arbitrarily sized requests into the same physical blocks.
A quick illustration of the memory waste with contiguous allocation:
# Assume max_seq_len=4096, block_size=16 for vLLM
contig_per_seq = 4096 * 2 # kv for llama-13b approx bytes per token omitted
paged_actual = 200 * 2 # a 200-token request only uses 200 tokens worth
waste_contig = contig_per_seq - paged_actual
print(f"wasted bytes per seq (contig): {waste_contig}")That wasted reservation is dead weight multiplied by every concurrent slot.
Scheduling granularity
vLLM’s scheduler runs every iteration, picking which sequences to prefill and which to decode. It can preempt low-priority sequences by swapping blocks to CPU. TGI’s rolling batch also does iteration-level scheduling, but its eviction story is weaker: it relies on pausing whole requests when memory pressure hits. Under heavy load that shows up as abrupt latency spikes rather than graceful degradation.
Latency at batch size 1
When you send a single request, the GPU is underutilized regardless of framework. The winner is the one with the smallest fixed cost per forward pass. TGI’s custom CUDA kernels for Llama and GPT-NeoX, combined with a thinner scheduling loop, typically return the first token a few milliseconds faster. vLLM pays a tax for its block manager and the abstraction of its model runner.
That tax is small in absolute terms—often lost in network jitter—but if you are building an interactive coding assistant where p50 latency matters more than p99 under load, TGI’s lean path is real.
# TGI launch for single-stream low latency
docker run --gpus all -p 8080:80 \
-e MAX_CONCURRENT_REQUESTS=1 \
-e MAX_BATCH_PREFILL_TOKENS=2048 \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id meta-llama/Llama-2-13b-chat-hf# vLLM equivalent, limiting concurrency
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-2-13b-chat-hf \
--max-num-seqs 1 \
--max-num-batched-tokens 2048Latency as batch size climbs
Push concurrent requests to 32, 64, or 128 and the batch size vllm vs tgi latency gap widens. vLLM’s paged blocks let it admit new sequences without reserving dead space for their maximum possible length. TGI’s contiguous cache means a 2K-token request and a 200-token request still consume slots sized by the router’s max. The result: vLLM sustains higher batch occupancy, keeping the GPU busy and holding p99 latency roughly flat. TGI starts queueing prefills because the prefill token budget is exhausted by a few long contexts.
A second effect is preemption. vLLM can block-swap a paused sequence’s KV pages to host RAM, decode the urgent one, then swap back. TGI’s all-or-nothing pause forces a full restart of the paused request’s prefill when memory frees. Under bursty traffic that difference decides whether your timeout budget survives.
Prefix caching adds another axis. vLLM’s automatic prefix caching reuses KV for shared system prompts; TGI has experimental support but it is less battle-tested. At large batch sizes with repeated prefixes, vLLM effectively shrinks the prefill token count per request, further flattening its latency curve.
Configuration knobs that expose the difference
Both servers expose batch controls, but they map to different internals.
# vLLM: hard cap on concurrent sequences and total tokens in flight
--max-num-seqs 256
--max-num-batched-tokens 8192# TGI: concurrent requests and prefill token budget per batch
--max-concurrent-requests 256
--max-batch-prefill-tokens 8192
--max-total-tokens 131072If you set --max-batch-prefill-tokens too low on TGI, large batches stall waiting for prefill slots. vLLM’s --max-num-batched-tokens interacts with paged blocks, so it degrades more smoothly: decode tokens keep flowing while prefill is chunked.
Measuring the curve yourself
Don’t trust a vendor chart. Stand up both containers on the same GPU and drive them with a fixed request distribution. Below is a minimal async loop using the OpenAI client against either endpoint (they are API-compatible).
import asyncio, openai, time
async def hit(client, prompt):
t0 = time.monotonic()
await client.completions.create(model="meta-llama/Llama-2-13b-chat-hf",
prompt=prompt, max_tokens=128)
return time.monotonic() - t0
async def sweep(concurrency, n):
client = openai.AsyncOpenAI(base_url="http://localhost:8080/v1")
sem = asyncio.Semaphore(concurrency)
async def limited(p):
async with sem:
return await hit(client, p)
prompts = ["Explain batch sizing." for _ in range(n)]
latencies = await asyncio.gather(*(limited(p) for p in prompts))
latencies.sort()
p99 = latencies[int(0.99*len(latencies))]
print(f"conc={concurrency} p99={p99:.3f}s")
# Run at 1, 8, 32, 128 concurrency
for c in [1, 8, 32, 128]:
asyncio.run(sweep(c, c*10))Run the sweep at increasing concurrency. Plot p99 versus concurrency. The curve for TGI will bend upward earlier; vLLM’s will stay shallow until you hit its --max-num-seqs wall. The exact crossing point depends on GPU memory and model size, but the shape is consistent.
Tradeoffs beyond latency
Throughput per dollar favors vLLM in almost every multi-tenant scenario because its memory efficiency raises batch occupancy. TGI still holds an edge for models with highly tuned kernels (e.g., certain BLOOM or GPT-NeoX variants) and for teams that want Rust-based serving with built-in tensor parallel sharding and a small binary footprint.
Operational complexity also differs. vLLM’s Python-centric stack integrates with PyTorch ecosystems and supports speculative decoding and quantization with less friction. TGI’s Rust/CUDA core is lighter to containerize but less flexible for custom model arches.
Routing at the gateway layer
If you put a gateway in front, you can exploit the divergence instead of fighting it. A gateway that honors client routing directives can send interactive single-shot requests to a TGI pool and bulk evaluation jobs to vLLM. n4n.ai, for instance, forwards provider cache-control hints and honors routing directives, so you can pin low-latency traffic to one backend and let high-batch traffic fall back to the other when a provider is degraded. That avoids rewriting app code when the latency curve shifts under you.
Decisive takeaway
Benchmark at the batch size you actually run, not the one in the README. If your p50 at batch size 1 is the product, TGI’s leaner path wins today. If you serve many users concurrently and care about p99 under burst, the batch size vllm vs tgi latency comparison lands firmly in vLLM’s favor because paged attention and fine-grained preemption keep the GPU saturated without tail explosions. Pick the framework that matches your concurrency profile, and keep the other as a fallback target.