Batching is the single most effective lever for increasing LLM serving throughput, but every batching decision pushes per-request latency in the opposite direction. Understanding the batching throughput vs latency tradeoff requires looking at how schedulers pack requests, how KV cache memory scales, and where your specific workload sits on the Pareto frontier. This post breaks down the mechanics, the math, and the practical decisions you’ll face when configuring a production inference stack.
The core tension
LLM inference is memory-bound, not compute-bound. The dominant cost is moving model weights and KV cache from HBM to the compute units. When you process one request at a time, the GPU sits mostly idle waiting on memory bandwidth. When you pack multiple requests into a single forward pass, you amortize the weight load across all requests in the batch — but every request in that batch must wait for the slowest one to finish its generation step.
This is the fundamental batching throughput vs latency tradeoff: throughput scales with batch size, but tail latency scales with the longest sequence in the batch.
Consider a simple example. A 7B model at FP16 needs ~14 GB for weights. Each token of KV cache consumes roughly 2 bytes per parameter per layer (key + value), so for 32 layers that’s ~448 KB per token. A batch of 32 requests with 2K context each needs ~28 GB just for KV cache — already exceeding a single A100 40GB. You hit memory limits before you hit compute saturation.
# Rough KV cache memory per request (bytes)
def kv_cache_bytes(num_layers, hidden_size, num_kv_heads, head_dim, seq_len, dtype_bytes=2):
# 2 * (key + value) * num_layers * num_kv_heads * head_dim * seq_len
return 2 * 2 * num_layers * num_kv_heads * head_dim * seq_len * dtype_bytes
# Llama-2-7B: 32 layers, 32 KV heads (GQA), 128 head_dim
# 2K context = ~28 MB per request
print(kv_cache_bytes(32, 4096, 32, 128, 2048) / 1e9) # ~28.7 GB for batch=32Static batching: simple but wasteful
Static batching pads all sequences in a batch to the same length and processes them in lockstep. The scheduler groups requests by similar prompt length, pads to the max, and runs a single forward pass per decode step.
# Static batching pseudocode
def static_batch_step(requests, max_batch_size, max_seq_len):
# Group by prompt length buckets
buckets = group_by_length(requests, bucket_size=256)
for bucket in buckets:
batch = bucket[:max_batch_size]
# Pad all to same length
padded = pad_sequences(batch, max_len=max_seq_len)
# Single forward pass
logits = model.forward(padded.input_ids, padded.attention_mask)
# All requests advance one token
return sample_next_token(logits)The problem: padding waste. If you batch a 100-token prompt with a 2000-token prompt, you spend 1900 steps computing on padding tokens for the short request. At 50% padding waste, you’re throwing away half your FLOPs. Static batching only makes sense when request lengths are naturally clustered — for example, a classified-ad generator where every prompt is ~200 tokens.
Continuous batching: the production standard
Continuous batching (also called iteration-level scheduling) removes padding waste by evicting finished requests and admitting new ones at every decode step. The batch composition changes dynamically. This is what vLLM, TGI, and TensorRT-LLM implement.
# Continuous batching scheduler loop (simplified)
class ContinuousBatchingScheduler:
def __init__(self, max_batch_tokens, max_num_seqs):
self.max_batch_tokens = max_batch_tokens
self.max_num_seqs = max_num_seqs
self.waiting = [] # RequestQueue
self.running = [] # List[Sequence]
self.kv_cache = KVCacheManager()
def step(self):
# 1. Evict finished sequences
self.running = [s for s in self.running if not s.is_finished()]
# 2. Admit new sequences while budget allows
while self.waiting and self.can_admit(self.waiting[0]):
seq = self.waiting.pop(0)
self.kv_cache.allocate(seq)
self.running.append(seq)
# 3. Single forward pass on all running sequences
if self.running:
input_ids = [s.next_token_id for s in self.running]
positions = [s.position for s in self.running]
logits = self.model.forward(input_ids, positions, self.kv_cache)
# 4. Sample and append
for seq, logit in zip(self.running, logits):
next_token = sample(logit)
seq.append_token(next_token)
return [s for s in self.running if s.is_finished()]
def can_admit(self, seq):
est_tokens = seq.prompt_len + seq.max_new_tokens
return (self.current_batch_tokens + est_tokens <= self.max_batch_tokens and
len(self.running) < self.max_num_seqs)The throughput win is real: continuous batching typically delivers 2-10x higher throughput than static batching at the same latency percentile, because GPU utilization stays near 100% without padding waste. But the latency story is nuanced.
Where latency hides
In continuous batching, a request’s latency depends on three factors:
- Queue time: Waiting for admission while the batch is full
- Decode time: Number of steps × step latency (shared across batch)
- Head-of-line blocking: Long requests delay admission of short ones behind them
The step latency itself grows with batch size because larger batches mean more KV cache reads/writes per step. On H100, a decode step for batch=1 might take 3 ms; at batch=128 it might take 12 ms. But you’re producing 128 tokens in that 12 ms instead of 1 token in 3 ms — throughput wins, but the per-request latency for a 500-token generation goes from 1.5s to 6s.
# Latency model for continuous batching
def estimate_latency(prompt_len, gen_len, batch_size, step_latency_fn):
# Queue time: depends on arrival rate and admission policy
queue_time = estimate_queue_time(batch_size, arrival_rate)
# Prefill: often chunked, but let's assume single prefill step
prefill_time = step_latency_fn(batch_size, prompt_len)
# Decode: gen_len steps at current batch size
# Batch size evolves as requests finish/arrive
decode_time = 0
current_batch = batch_size
for i in range(gen_len):
decode_time += step_latency_fn(current_batch, 1)
# Simulate some requests finishing
current_batch = max(1, current_batch - poisson(0.02 * current_batch))
return queue_time + prefill_time + decode_timeScheduling policies change the curve
How you admit requests shapes the latency distribution. Three common policies:
First-come-first-served (FCFS): Simple, fair, but a single long request can block many short ones. Bad for tail latency.
Shortest-remaining-time-first (SRTF): Prioritize requests with fewest tokens left. Minimizes average latency but starves long requests. Requires knowing remaining length (easy for fixed max_tokens, hard for stop-sequence conditions).
Preemptive priority with length buckets: Group requests into length buckets (short/medium/long). Serve short buckets with strict latency SLOs, fill remaining capacity with long requests. This is what production systems like vLLM’s --scheduling-policy priority approximate.
# Priority bucket scheduler sketch
class PriorityBucketScheduler:
def __init__(self):
self.buckets = {
'short': RequestQueue(max_tokens=512), # SLO: p99 < 2s
'medium': RequestQueue(max_tokens=2048), # SLO: p99 < 10s
'long': RequestQueue(max_tokens=8192), # Best effort
}
self.capacity = {'short': 0.5, 'medium': 0.3, 'long': 0.2}
def admit(self, max_batch_tokens):
admitted = []
for bucket_name, quota in self.capacity.items():
bucket = self.buckets[bucket_name]
budget = int(max_batch_tokens * quota)
while bucket and budget >= bucket.peek_estimated_tokens():
admitted.append(bucket.pop())
budget -= admitted[-1].estimated_tokens
return admittedMemory management: the hidden constraint
KV cache is the scarcest resource. Continuous batching needs a paged KV cache (vLLM’s key innovation) to avoid fragmentation. Instead of contiguous allocation per request, you allocate fixed-size blocks (typically 16 or 32 tokens) and map logical positions to physical blocks.
# Paged KV cache allocation
class PagedKVCache:
def __init__(self, num_blocks, block_size, num_layers, num_kv_heads, head_dim):
self.block_size = block_size
self.num_blocks = num_blocks
# [num_layers, 2, num_blocks, block_size, num_kv_heads, head_dim]
self.kv_cache = torch.empty(
num_layers, 2, num_blocks, block_size, num_kv_heads, head_dim,
dtype=torch.float16, device='cuda'
)
self.free_blocks = list(range(num_blocks))
self.block_tables = {} # seq_id -> List[block_id]
def allocate(self, seq_id, num_tokens):
num_blocks = (num_tokens + self.block_size - 1) // self.block_size
if len(self.free_blocks) < num_blocks:
raise OOMError("KV cache full")
blocks = [self.free_blocks.pop() for _ in range(num_blocks)]
self.block_tables[seq_id] = blocks
return blocks
def free(self, seq_id):
self.free_blocks.extend(self.block_tables.pop(seq_id))
def get_block_table(self, seq_id):
return self.block_tables[seq_id]Paged attention lets you pack variable-length sequences tightly. But it adds an indirection: every attention kernel must gather KV from scattered blocks. The kernel overhead is ~5-10% vs contiguous cache — a worthwhile tradeoff for the memory efficiency.
Prefill vs decode: different batching regimes
Prefill (prompt processing) and decode (token generation) have fundamentally different characteristics:
| Phase | Compute pattern | Batching behavior |
|---|---|---|
| Prefill | Matrix-matrix (prompt × weights) | Large batches, high arithmetic intensity |
| Decode | Matrix-vector (1 token × weights) | Small batches, memory-bound |
Smart systems chunk long prefill into multiple forward passes to avoid OOM and keep decode slots available. vLLM’s --max-num-batched-tokens controls the combined prefill+decode token budget per step.
# Chunked prefill example
def chunked_prefill(model, input_ids, max_batch_tokens, kv_cache):
seq_len = input_ids.shape[1]
chunk_size = min(max_batch_tokens, 2048) # Tunable
for start in range(0, seq_len, chunk_size):
end = min(start + chunk_size, seq_len)
chunk = input_ids[:, start:end]
# Forward with causal mask allowing attention to past KV
logits = model.forward(chunk, kv_cache, start_pos=start)
kv_cache.append(chunk) # Update KV cache incrementally
return logits[:, -1:] # Return last position for decode startThis lets you serve a 32K context request on a GPU that only has KV budget for 4K tokens per step — at the cost of higher prefill latency for that request.
Quantization and batching interaction
Quantization changes the batching math. INT4 weights reduce model size by 4x vs FP16, but KV cache stays FP16 (or FP8) because quantizing KV degrades quality noticeably. This means:
- Weight memory drops → more room for KV cache → larger batches possible
- Memory bandwidth drops proportionally → decode step latency improves
- But dequantization overhead adds compute — on Hopper, FP4/FP8 tensor cores make this nearly free; on Ampere, it’s a measurable penalty
# Rough memory budget for Llama-3-8B on H100 80GB
# FP16: 16 GB weights + KV cache for ~150K tokens
# INT4 (AWQ/GPTQ): 4 GB weights + KV cache for ~350K tokens
# FP8 (native): 8 GB weights + KV cache for ~250K tokensWith INT4, you can run batch=256 decode on a single H100 where FP16 maxes at batch=80. The throughput gain is real, but each request now waits behind 255 others — median latency triples even as throughput quadruples.
When to choose what
High throughput, latency-tolerant (batch inference, async workloads, offline processing):
- Maximize batch size
- Use continuous batching with large
max_batch_tokens - Enable chunked prefill for long contexts
- Quantize to INT4/FP8
- Accept p99 latency of 10-60s
Low latency, throughput-secondary (chat, real-time agents, user-facing):
- Cap batch size aggressively (
max_num_seqs=16-32) - Use priority scheduling with short-request buckets
- Keep KV cache in FP16/FP8 (avoid INT4 KV)
- Consider speculative decoding for latency reduction
- Target p99 < 2s for short requests
Mixed workload (most production systems):
- Run two model replicas: one tuned for throughput, one for latency
- Route by request type:
max_tokens < 512→ latency replica; else → throughput replica - Or use a single replica with priority buckets and strict admission control
# Routing logic example
def route_request(request, latency_replica, throughput_replica):
if request.max_tokens <= 512 and request.priority == 'high':
return latency_replica
elif request.estimated_tokens > 4096:
return throughput_replica # Long requests go to high-batch replica
else:
# Load-based routing
if latency_replica.current_load < 0.7:
return latency_replica
return throughput_replicaThe decisive takeaway
Batching is not a knob you turn to “more throughput” — it’s a structural decision that defines your latency distribution. Continuous batching with paged KV cache is the correct default for virtually all production LLM serving. But the parameters of that batching (max batch tokens, max sequences, scheduling policy, prefill chunking) must be chosen based on your actual SLOs, not theoretical peak throughput.
Measure your workload’s prompt/gen length distribution. Set max_batch_tokens to the 95th percentile of (prompt + max_tokens) × target concurrency. Use priority buckets if you have latency SLOs. Quantize weights aggressively but keep KV cache in FP16/FP8. And always, always run load tests that capture the tail — average latency lies.
The batching throughput vs latency tradeoff doesn’t disappear with better kernels or faster GPUs. It’s inherent to the memory-bound nature of autoregressive generation. Your job is to pick the right point on the curve for each workload, not to pretend the curve doesn’t exist.