Static batching wastes GPU cycles. When a request finishes early, its slot sits idle until the entire batch completes. Continuous batching gpu utilization solves this by evicting finished sequences and admitting new ones on every iteration, keeping the device saturated. The thesis is straightforward: continuous batching improves throughput by 2-4x over static batching for typical LLM workloads, but it shifts the bottleneck from compute to scheduler overhead and KV cache management. If you’re serving LLMs at scale, you need to understand exactly where those new bottlenecks live.
The problem with static batching
Static batching pads every sequence to the same length and processes them in lockstep. A batch of 32 requests with max length 2048 tokens means every forward pass computes 32 × 2048 positions, even if 30 of those requests finished at 500 tokens. The GPU computes on padding tokens that contribute nothing to throughput.
# Static batching: wasted compute on padding
batch_size = 32
max_len = 2048
# Actual work: sum of real lengths
# Wasted work: batch_size * max_len - sum(real_lengths)The utilization curve looks like a sawtooth. As requests complete, active token count drops but the kernel launch configuration stays fixed. You’re launching a 32×2048 GEMM when only 8×500 tokens need computation. The arithmetic intensity collapses.
How continuous batching works
Continuous batching treats the batch as a fluid. On each iteration, the scheduler:
- Removes sequences that hit EOS or max length
- Appends new requests from the queue
- Runs a single forward pass on the concatenated token stream
- Returns logits for every active position
# Conceptual continuous batching loop
active_seqs = [] # list of (seq_id, token_ids, kv_cache_ptrs)
waiting_queue = deque()
while active_seqs or waiting_queue:
# Admit new requests up to memory budget
while waiting_queue and can_admit(waiting_queue[0]):
seq = waiting_queue.popleft()
active_seqs.append(init_sequence(seq))
# Single forward pass on all active tokens
logits = model.forward(concat_tokens(active_seqs))
# Sample next tokens, update KV caches
for i, seq in enumerate(active_seqs):
next_token = sample(logits[seq.offset : seq.offset + 1])
seq.append(next_token)
if seq.is_finished():
evict(seq)The kernel sees a single 1D tensor of shape [total_active_tokens, hidden_dim] instead of [batch, seq_len, hidden_dim]. This is the key: the GEMM dimensions stay large even as individual sequences come and go.
Memory pressure and KV cache dynamics
Continuous batching gpu utilization lives or dies by KV cache management. Each active sequence holds 2 * num_layers * num_heads * head_dim * seq_len bytes of KV cache. With 32-layer models and 4096 context, that’s ~1.5 GB per sequence at FP16. You cannot admit unlimited requests.
The scheduler must track per-sequence cache allocation and support:
- Variable-length sequences: KV cache is a jagged 2D array, not a dense tensor
- Cache eviction: Free blocks when sequences finish
- Fragmentation: Non-contiguous free blocks after many evictions
# Block-based KV cache manager (simplified)
class KVCacheManager:
def __init__(self, num_blocks, block_size, num_layers, num_heads, head_dim):
self.free_blocks = list(range(num_blocks))
self.allocated = {} # seq_id -> list[block_ids]
self.block_size = block_size
def allocate(self, seq_id, num_tokens):
needed = (num_tokens + self.block_size - 1) // self.block_size
if len(self.free_blocks) < needed:
return None # OOM
blocks = self.free_blocks[:needed]
self.free_blocks = self.free_blocks[needed:]
self.allocated[seq_id] = blocks
return blocks
def free(self, seq_id):
self.free_blocks.extend(self.allocated.pop(seq_id))Real systems like vLLM use PagedAttention with 16-32 token blocks. This reduces fragmentation but adds indirection: the attention kernel must gather KV blocks via an indirection table. That gather costs cycles. At high batch sizes, the indirection overhead becomes measurable.
Scheduling policies that matter
Admission policy determines whether you maximize throughput or minimize latency. Three common policies:
First-come-first-served (FCFS) admits requests in arrival order. Simple, fair, but head-of-line blocking hurts tail latency when a long request occupies cache blocks.
Shortest-job-first (SJF) prioritizes requests with fewer remaining tokens. Improves average latency but starves long requests. Requires estimating remaining length — often unknown for open-ended generation.
Chunked prefill splits long prefill across multiple iterations. Instead of admitting a 4096-token prefill that monopolizes the GPU for 200ms, you process 512 tokens per iteration, interleaving with decode work from other requests.
# Chunked prefill admission
def can_admit(req, max_tokens_per_iter=512):
if req.is_prefill:
return estimate_prefill_chunks(req) * max_tokens_per_iter <= budget
return req.remaining_tokens <= budgetChunked prefill is the practical sweet spot. It prevents a single massive prefill from starving decode work, which is where continuous batching gpu utilization shines — decode is memory-bound and benefits most from large batch sizes.
Tradeoffs you actually face
Continuous batching is not free. Here are the real costs:
Scheduler overhead: Every iteration runs Python (or C++) logic to manage queues, allocate cache blocks, and construct the attention mask. At 1000+ tokens/sec, this overhead competes with kernel launch latency. vLLM moves the scheduler to a dedicated thread; TensorRT-LLM uses a C++ executor. You need the scheduler faster than your slowest kernel.
Variable-length attention: The attention mask is no longer a clean triangular matrix. You need either a block-sparse kernel (complex) or a padded mask with explicit sequence boundaries (wastes compute on cross-sequence attention). FlashAttention-2 supports variable-length sequences via cu_seqlens, but the kernel launch config changes every iteration.
Cache fragmentation: Even with paged allocation, long-running sequences create “islands” of allocated blocks. A new 2048-token request may fail admission despite 50% free cache because no contiguous run of blocks exists. Compaction (copying KV cache) costs bandwidth and stalls the GPU.
Priority inversion: A high-priority short request waits behind a low-priority long prefill. Solutions exist — preemption, priority queues — but add complexity. Most production systems accept FCFS with chunked prefill as “good enough.”
Observability gaps: Static batching gives you clean per-request latency percentiles. Continuous batching interleaves work; a request’s latency depends on what else was in the batch at each step. You need per-iteration logging to debug tail latency.
When static batching still wins
Continuous batching gpu utilization assumes variable-length generation. If your workload is fixed-length — classification, embedding, reranking — static batching is simpler and faster. No scheduler, no fragmentation, no variable-length kernels. The padding waste is bounded and predictable.
Similarly, if you serve a single model with batch sizes that saturate the GPU (e.g., 128+ concurrent requests of similar length), static batching utilization approaches 90%+. The marginal gain from continuous batching may not justify the operational complexity.
Decisive takeaway
Adopt continuous batching for any open-ended generation workload with variable output lengths. The throughput gains are real and compound at scale. But treat the scheduler and KV cache manager as critical infrastructure — not glue code. Invest in a block-based cache allocator, chunked prefill, and a scheduler thread that never blocks the GPU. If you’re building this yourself, start with vLLM’s architecture as a reference; if you’re buying, verify the vendor’s scheduler handles fragmentation and priority inversion without manual tuning. The GPU is the expensive part; the scheduler’s job is to keep it fed.