Batching in LLM inference is the practice of grouping multiple independent requests so they execute together on the same GPU kernels, amortizing the fixed cost of model weight loading and matrix multiplication setup across many tokens. Without batching, each request would launch its own kernel sequence, leaving compute units underutilized while memory bandwidth sits idle. The core tension is that larger batches increase throughput but also increase per-request latency, since every request in the batch must wait for the slowest one to finish its current step.
How batching works at the kernel level
When a transformer model runs inference, the dominant operation is matrix multiplication: the input token embeddings (batch_size × seq_len × hidden_dim) multiply against weight matrices (hidden_dim × hidden_dim or hidden_dim × vocab_size). GPUs achieve high utilization only when these matrices are large enough to saturate all streaming multiprocessors. A single request with sequence length 1 produces a matrix of shape (1 × hidden_dim) — far too small. By stacking 32, 64, or 128 requests together, the batch dimension grows and the GEMM kernels approach peak FLOPS.
The attention mechanism adds a wrinkle. Each request maintains its own key-value (KV) cache — past keys and values for every layer, shape (num_layers × 2 × batch_size × num_heads × seq_len × head_dim). In static batching, all requests in a batch must have the same sequence length, so the KV cache tensors are dense and contiguous. This makes memory access predictable but forces padding or truncation.
# Static batching: all sequences padded to max_len
batch_input_ids = torch.full((batch_size, max_len), pad_token_id)
for i, seq in enumerate(requests):
batch_input_ids[i, :len(seq)] = torch.tensor(seq)
# KV cache allocated once for the whole batch
kv_cache = torch.zeros(
num_layers, 2, batch_size, num_heads, max_len, head_dim,
dtype=torch.float16, device="cuda"
)Static batching vs continuous batching
Static batching waits until a batch is full (or a timeout fires), then processes all requests to completion before accepting new ones. Simple to implement, but it creates head-of-line blocking: a single long request holds the entire batch, and short requests that finish early still occupy their slots until the batch drains.
Continuous batching (also called iteration-level scheduling) removes finished requests from the batch mid-generation and admits new requests in the same iteration. The KV cache becomes a dynamic data structure — typically a paged allocation scheme where each request owns a variable-length list of physical blocks. The scheduler compacts the batch each step, keeping GPU utilization high while reducing average latency.
# Continuous batching: dynamic batch size per iteration
class ContinuousBatcher:
def __init__(self, max_batch_size, max_seq_len, block_size=16):
self.max_batch_size = max_batch_size
self.block_manager = BlockManager(max_seq_len, block_size)
self.running = [] # list of Request objects
self.waiting = deque()
def step(self):
# 1. Remove finished requests, free their blocks
self.running = [r for r in self.running if not r.finished]
for r in self.running:
if r.finished:
self.block_manager.free(r.blocks)
# 2. Admit new requests up to max_batch_size
while len(self.running) < self.max_batch_size and self.waiting:
req = self.waiting.popleft()
req.blocks = self.block_manager.allocate(req.prompt_len + 1)
self.running.append(req)
# 3. Build contiguous tensors for this iteration
input_ids = torch.cat([r.next_token for r in self.running])
positions = torch.cat([r.position for r in self.running])
block_tables = torch.stack([r.block_table for r in self.running])
# 4. Single forward pass with paged attention kernel
logits = model(input_ids, positions, block_tables, kv_cache)
# 5. Sample next tokens, update request state
for i, r in enumerate(self.running):
r.next_token = sample(logits[i])
r.position += 1
if r.position >= r.max_tokens or r.next_token == eos_token:
r.finished = TruePaged attention (introduced in vLLM) is the key enabler. Instead of a monolithic KV tensor, physical memory is divided into fixed-size blocks (typically 16 or 32 tokens). Each request maintains a logical block table mapping its sequence positions to physical blocks. The attention kernel reads keys and values by following this indirection, allowing non-contiguous, variable-length sequences to share the same batch.
Why batching matters for throughput and cost
The arithmetic intensity of transformer inference — FLOPS per byte of memory traffic — is low during the decode phase. For a 7B parameter model, each generated token reads ~14 GB of weights (2 bytes per parameter × 7B) but performs only ~14 GFLOPs of compute. Memory bandwidth, not compute, is the bottleneck. Batching improves arithmetic intensity by reusing the same weight bytes across many requests: the weight matrix stays in cache or is streamed once while the input matrix grows with batch size.
Roofline model analysis shows the crossover point. On an H100 (3 TB/s HBM3, 1979 TFLOPS FP16), the ridge point is ~660 FLOPS/byte. Single-request decode operates at ~1 FLOPS/byte — deep in the memory-bound region. At batch size 32, arithmetic intensity rises to ~32 FLOPS/byte, still memory-bound but 32× closer to the ridge. At batch size 256, you approach compute saturation.
The economic implication is direct: a GPU running at batch size 1 might serve 50 tokens/second total. The same GPU at batch size 64 serves 2,000+ tokens/second. If you pay $2.50/hour for that GPU, the per-million-token cost drops from $50 to ~$1.25. This is why every production inference engine (vLLM, TGI, TensorRT-LLM, SGLang) implements continuous batching as a baseline feature.
Concrete example: serving a chat endpoint
Consider a /v1/chat/completions endpoint receiving requests with varying prompt lengths and max_tokens. Without batching, each request spins up a separate model invocation. With static batching (batch_size=8, timeout=100ms), the server accumulates requests, pads all prompts to the longest in the batch, and generates in lockstep. A request with 100 prompt tokens and max_tokens=10 waits for a request with 2000 prompt tokens and max_tokens=500 — the short request’s GPU time is wasted on padding.
With continuous batching, the server admits 8 requests. After 10 steps, the short request finishes and its KV blocks are freed. Two new requests enter the batch. The long request continues uninterrupted. The GPU processes 8 requests worth of tokens every step, but the batch composition changes dynamically. Average latency drops because short requests don’t queue behind long ones; throughput rises because the batch rarely has empty slots.
// Request A: short
{"model": "llama-3-8b", "messages": [{"role": "user", "content": "Hi"}], "max_tokens": 10}
// Request B: long
{"model": "llama-3-8b", "messages": [{"role": "user", "content": "Summarize this 50-page doc: ..."}], "max_tokens": 2000}
// Static batching: both wait for 2000 steps
// Continuous batching: A finishes at step 10, B continues alone or with new requestsCommon misconceptions
Misconception: “Batching only helps throughput, not latency.”
Continuous batching improves average latency and tail latency compared to static batching, because short requests exit early. However, per-request latency at a given batch size is higher than a dedicated GPU would provide — the request shares compute with others. The tradeoff is latency per request vs. cost per token. For user-facing chat, target batch sizes that keep p50 latency under 200ms; for background workloads, push batch size to maximize throughput.
Misconception: “Larger batch size is always better.”
Past the compute saturation point, larger batches increase latency without improving throughput. They also increase KV cache memory pressure, risking OOM. The optimal batch size depends on model size, sequence length, and GPU memory. Profile with your actual workload; don’t copy numbers from benchmarks.
Misconception: “Continuous batching requires paged attention.”
You can implement continuous batching with a monolithic KV cache by reallocating and copying on every iteration — but the copy overhead kills the benefit at scale. Paged attention (or similar block-based schemes) is practically mandatory for production continuous batching. The block size (16, 32, 64) trades off internal fragmentation against block table size and kernel launch overhead.
Misconception: “Batching works the same for prefill and decode.”
Prefill (processing the prompt) is compute-bound: large matrix multiplies with sequence length > 1. Decode (generating one token per step) is memory-bound. Continuous batching typically mixes prefill and decode in the same batch — new requests enter at prefill, existing requests are in decode. The scheduler must account for the different compute/memory profiles. Some engines (SGLang, vLLM with chunked prefill) split prefill into chunks to avoid long prefill requests starving decode.
Misconception: “My framework handles batching automatically.”
Frameworks provide the primitives, but you still choose: max_batch_size, max_num_seqs, max_tokens_per_batch, prefill chunk size, block size, scheduling policy (FCFS, priority, deadline-aware). These knobs interact. A misconfigured max_num_seqs can cause OOM before max_batch_size is reached. A too-small prefill chunk size increases kernel launch overhead. Treat batching configuration as part of your capacity planning, not a set-and-forget default.
Capacity planning with batching
When sizing a deployment, work backward from your SLOs. Suppose you need p99 latency < 500ms for 100 concurrent users, each generating 500 tokens at 20 tokens/second. That’s 25 seconds of generation per request, 2500 token-seconds of work per request, 250,000 token-seconds total. An H100 delivers ~3000 tokens/second at batch 64 for Llama-3-8B. You need ~83 GPU-seconds, or ~3.5 GPUs for the burst. But concurrency is bursty — provision for peak concurrent requests × average tokens per request / (throughput per GPU × target utilization).
def estimate_gpus(concurrent_users, avg_output_tokens, tokens_per_sec_per_gpu, target_util=0.7):
total_token_seconds = concurrent_users * avg_output_tokens
gpu_seconds = total_token_seconds / tokens_per_sec_per_gpu
return math.ceil(gpu_seconds / target_util)
# Example: 200 concurrent, 800 tokens each, 2500 tok/s/GPU (batch 32, 7B model)
# estimate_gpus(200, 800, 2500) -> 92 GPU-seconds -> 2 GPUs at 70% util for 60s burstMonitor batch size distribution, queue depth, and KV cache hit rate in production. If batch size frequently hits max_batch_size but GPU utilization is low, the bottleneck is scheduler overhead or CPU-side tokenization. If KV cache utilization exceeds 90%, you’re one traffic spike from OOM — enable prefix caching or reduce max_num_seqs.
Summary
Batching transforms LLM inference from memory-bound single-stream execution into a throughput-oriented batched workload. Static batching is simple but wastes capacity on padding and head-of-line blocking. Continuous batching with paged attention is the production standard: it keeps GPUs saturated, reduces tail latency, and enables multi-tenant serving on shared infrastructure. The key parameters — max batch size, block size, prefill chunking, scheduling policy — require tuning for your model, hardware, and latency targets. Treat batching configuration as a first-class operational concern, not an implementation detail.