Batch size vs latency: the inference trade-off

The batch size vs latency trade-off is the single most important lever in LLM serving. Every inference engine — vLLM, TGI, TensorRT-LLM — exposes it, but the documentation rarely explains what actually happens when you crank the number up or down. This post breaks down the mechanics, the memory pressure, and the operational reality so you can pick a batching strategy that matches your workload.

What batch size means for LLM inference

In LLM serving, batch size is the number of independent requests processed simultaneously by the same model instance. Unlike training, where batches are fixed tensors, inference batches are dynamic: requests arrive at different times, have different prompt lengths, and generate different numbers of tokens.

When you set max_batch_size=32, you’re telling the scheduler: “pack up to 32 requests into one forward pass.” The engine pads shorter sequences to the longest prompt in the batch, runs one matrix multiply, then splits the logits back out. This amortizes the model’s weight-read cost across all requests — the fundamental throughput win.

But there’s a catch. Padding wastes compute on <pad> tokens. A batch with one 4k-token prompt and thirty-one 100-token prompts spends most of its FLOPs on padding. Smart schedulers (vLLM’s chunked prefill, TGI’s padded batching) mitigate this, but the waste never fully disappears.

# vLLM example: static batching config
from vllm import SamplingParams, LLM

llm = LLM(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    max_num_seqs=32,           # max concurrent requests
    max_num_batched_tokens=8192,  # token budget per forward pass
    enable_chunked_prefill=True,  # split long prefill across iterations
)

The latency-throughput curve

The batch size vs latency curve has three regimes. Understanding which regime you’re in determines whether increasing batch size helps or hurts.

Regime 1: Under-utilized (batch size 1–4). The GPU sits idle between requests. Time-to-first-token (TTFT) is minimal because there’s no queueing, but throughput is abysmal — often 10–20% of peak. This is the default for naive deployments.

Regime 2: Sweet spot (batch size 8–64 depending on model/hardware). The compute units stay saturated. TTFT grows linearly with queue depth, but throughput approaches hardware limits. For Llama-3-70B on H100, the sweet spot often lands around 16–32 concurrent sequences.

Regime 3: Memory-bound (batch size >64). KV cache eats all VRAM. The engine starts swapping to CPU or rejecting requests. TTFT spikes unpredictably. Throughput plateaus or drops.

# Quick sanity check: measure TTFT vs batch size locally
for bs in 1 2 4 8 16 32 64; do
  python benchmark.py --batch-size $bs --model meta-llama/Meta-Llama-3-8B-Instruct
done

Memory and KV cache implications

KV cache is the hidden variable in the batch size vs latency equation. Every token in every active sequence consumes 2 * num_layers * hidden_size * sizeof(dtype) bytes. For Llama-3-70B (80 layers, 8192 hidden, bfloat16), that’s ~2.1 MB per 1k tokens per sequence.

At batch size 32 with 4k average context: 32 * 4 * 2.1 MB ≈ 270 GB. You need 4×H100 (80 GB each) just for KV cache, before model weights.

This is why max_num_batched_tokens matters more than max_num_seqs. The token budget caps total KV cache per forward pass, letting you trade sequence count for context length.

# Memory-aware batching: cap tokens, not sequences
llm = LLM(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    max_num_seqs=256,              # allow many short requests
    max_num_batched_tokens=16384,  # but limit total tokens/step
    gpu_memory_utilization=0.9,    # leave headroom for KV cache growth
)

Continuous batching vs static batching

Static batching waits until a batch fills or a timeout fires, then runs one forward pass. Simple, but wastes slots on short requests that finish early while long requests keep generating.

Continuous batching (also called iteration-level scheduling) evicts finished sequences each step and admits new ones. The batch composition changes every iteration. This keeps GPU utilization high without padding waste.

vLLM, TGI, and TensorRT-LLM all implement continuous batching now. The difference is in the scheduler policy: vLLM uses a prefix-aware chunked prefill; TGI uses a token-budget queue with priority; TensorRT-LLM uses a centralized executor with inflight batching.

# TGI continuous batching config (launcher args)
# --max-batch-prefill-tokens 8192
# --max-batch-total-tokens 16384
# --max-concurrent-requests 128
# --max-waiting-tokens 20

The latency profile differs: static batching has predictable TTFT (timeout-bound), continuous batching has variable TTFT (queue-bound). For latency-sensitive paths, you often want a dedicated small-batch pool.

Comparison: batching strategies across dimensions

Which to choose by use case

Chat assistants, coding copilots, interactive agents. Target TTFT < 200 ms. Run a dedicated pool with max_num_seqs=4–8 and aggressive prefill chunking. Accept lower throughput; latency is the product. If you’re running n4n.ai or a similar gateway, route these requests to a low-batch-size model instance.

Batch document processing, embedding generation, eval pipelines. Maximize throughput. Push max_num_batched_tokens to VRAM limit, enable continuous batching, let the scheduler pack thousands of tokens per step. TTFT doesn’t matter — only tokens/sec/dollar.

Mixed workloads (most production systems). Run two pools: a latency pool (batch 4–8) and a throughput pool (batch 32–128). Route by priority header or request type. The gateway layer handles this cleanly — one endpoint, internal routing based on x-batch-priority or similar.

Streaming with strict per-token latency budgets. Consider speculative decoding or draft models instead of batch size tuning. Batch size helps throughput, but per-token decode latency is memory-bandwidth bound. A smaller model drafting for a larger one often beats any batching trick.

Multi-LoRA serving. Each adapter adds KV cache overhead. Continuous batching with per-adapter token budgets prevents one heavy adapter from starving others. Set max_loras=16 and max_lora_rank=64 as starting points, then measure.

Operational guardrails

Don’t guess — instrument. Export these metrics from your inference server:

• ttft_p50, ttft_p99 by batch size bucket
• tokens_per_second per GPU
• kv_cache_usage_bytes over time
• queue_depth and rejection_rate

# Prometheus queries to watch
histogram_quantile(0.99, rate(vllm_ttft_seconds_bucket[5m]))
rate(vllm_generation_tokens_total[5m]) / count(vllm_gpu_memory_used_bytes)
vllm_kv_cache_usage_bytes / vllm_kv_cache_capacity_bytes

When kv_cache_usage > 0.85 * capacity, you’re in the danger zone. Scale out or drop max_num_batched_tokens.

The bottom line

Batch size vs latency isn’t a single knob — it’s a multi-dimensional constraint surface. The right answer depends on your TTFT SLO, your GPU fleet, your prompt length distribution, and whether you’re serving one model or fifty. Start with continuous batching, a token budget at 80% VRAM, and a latency pool for interactive traffic. Measure. Adjust. Repeat.