Static batching vs continuous batching, compared

Static batching and continuous batching represent two fundamentally different approaches to scheduling LLM inference requests. Static batching groups requests into fixed-size batches that execute to completion before the next batch begins. Continuous batching — sometimes called iteration-level scheduling — admits new requests as soon as GPU memory frees up, interleaving prefill and decode steps across requests. The choice between them determines your tail latency, GPU utilization, and how much engineering effort you spend on the serving stack.

How static batching works

Static batching collects incoming requests until a batch reaches a configured size or a timeout expires. The entire batch then runs through prefill (processing all input tokens in parallel) followed by decode (generating tokens autoregressively) until every sequence hits its stop condition or max length. Only then does the next batch start.

# conceptual static batching loop
while True:
    batch = collect_requests(batch_size=32, timeout_ms=10)
    if not batch:
        continue
    prefill(batch)           # all inputs processed together
    while not all_done(batch):
        decode_step(batch)   # generate one token per sequence
    return_results(batch)

The scheduler is simple: a FIFO queue, a batch size knob, and a timeout. No preemption, no dynamic memory management during generation. This simplicity is why vLLM’s initial release used static batching, and why many custom inference engines still do.

How continuous batching works

Continuous batching treats each decoding iteration as a scheduling opportunity. When a sequence finishes (hits EOS or max tokens), its KV cache blocks are freed immediately. The scheduler can then admit a waiting request — running its prefill — in the very next iteration, alongside the ongoing decode steps of other sequences.

# conceptual continuous batching loop
waiting_queue = []
running_batch = []

while True:
    # admit new requests if memory allows
    while waiting_queue and can_fit_prefill(waiting_queue[0]):
        req = waiting_queue.pop(0)
        running_batch.append(start_prefill(req))

    # single decode step for all running sequences
    decode_step(running_batch)

    # evict finished sequences, reclaim KV blocks
    running_batch = [seq for seq in running_batch if not seq.finished]

This requires a block manager (like vLLM’s PagedAttention) that allocates KV cache in fixed-size blocks rather than contiguous tensors. The scheduler becomes a memory allocator: it tracks free blocks, estimates prefill memory for incoming requests, and packs them alongside active decodes.

Latency and throughput comparison

Static batching’s TTFT variance comes from the timeout: a request arriving just after a batch starts waits nearly the full timeout plus the batch’s prefill time. Continuous batching admits requests as soon as memory permits, but a large prefill can still stall decode for existing sequences if the scheduler lacks priority controls.

Throughput favors continuous batching in most real workloads. Static batching leaves GPU cycles on the table during the “tail” of a batch — when only a few sequences remain active but the whole batch still occupies memory. Continuous batching reclaims that memory immediately. The gap widens with variable output lengths: a single 4k-token sequence in a static batch of 32 holds the GPU hostage while 31 sequences finished at 100 tokens.

Memory and KV cache management

Static batching allocates a single contiguous KV tensor of shape [batch_size, num_layers, 2, max_seq_len, num_heads, head_dim]. Every sequence in the batch reserves max_seq_len slots regardless of actual length. A batch of 32 sequences with 4k max length at bfloat16 (LLaMA-7B, 32 layers, 32 heads, 128 dim) consumes roughly 32 × 32 × 2 × 4096 × 32 × 128 × 2 bytes ≈ 17 GB — just for KV cache.

Continuous batching with PagedAttention splits KV cache into blocks (typically 16 or 32 tokens per block). A 4k sequence uses 128 blocks of 32 tokens; a 100-token sequence uses 4 blocks. Free blocks return to a global pool. The same 32-sequence workload with mixed lengths might use 4–6 GB. This is the single biggest operational advantage: you fit 3–4× more concurrent requests on the same GPU.

The trade-off is indirection. PagedAttention requires a block table lookup per attention head per layer per token. Modern kernels (FlashAttention-2/3, PagedAttention kernels) amortize this well, but it adds kernel launch overhead and complicates prefix caching — shared prefix blocks must be reference-counted and protected from eviction.

Prefill-decode interference

Continuous batching interleaves prefill and decode in the same iteration. Prefill is compute-bound (large matrix multiplies); decode is memory-bound (small matmuls, KV cache reads). When a large prefill enters the batch, it can starve decode steps of compute resources, increasing per-token latency for in-flight requests.

Mitigations exist but add complexity:

• Chunked prefill: split large prefills across multiple iterations, limiting tokens processed per step.
• Priority scheduling: decode steps get compute priority; prefill only runs when decode occupancy is low.
• Separate prefill/decode pools: dedicate SMs or even separate GPUs to prefill (disaggregated serving).

Static batching avoids this entirely by separating phases. If your workload is prefill-heavy (short outputs, long contexts) and latency-sensitive, static batching’s predictability can outweigh its throughput penalty.

Ergonomics and ecosystem

Static batching is easier to implement correctly. The mental model maps directly to batch matrix multiplication. Debugging is straightforward: a batch either succeeds or fails as a unit. Most custom inference engines (TensorRT-LLM’s basic executor, older Triton backends) default to static batching for this reason.

Continuous batching requires a block manager, a scheduler with admission control, and careful handling of:

• Prefill chunking boundaries
• KV block eviction policies (LRU, FIFO, priority-aware)
• Prefix caching integration
• Speculative decoding coordination (draft and target models must share block tables)

vLLM, TGI (Text Generation Inference), and SGLang have production-hardened continuous batching. If you build on these, you inherit the complexity. If you’re writing a custom engine — say, for a specialized model architecture or hardware target — continuous batching is a significant engineering investment.

n4n.ai’s gateway layer sits above the inference engine and sees the practical impact: continuous batching backends sustain higher request rates per GPU, which translates to fewer replicas for the same throughput target. But the gateway also handles routing directives that work with either batching strategy — the client doesn’t need to know which scheduler runs underneath.

When to choose static batching

• Fixed, predictable workloads: batch inference jobs, evaluation pipelines, offline processing where latency doesn’t matter.
• Simple deployment constraints: you need a minimal dependency footprint, no PagedAttention kernel support on your hardware (some older GPUs, custom ASICs).
• Prefill-dominated, latency-sensitive: every request is a long-context prefill with short output (e.g., RAG with 32k context, 50-token answer). Phase separation prevents prefill from jittering decode.
• Prototyping and debugging: you want to verify model correctness before investing in a complex scheduler.

When to choose continuous batching

• Online serving with variable output lengths: chat, code generation, agents — anywhere some responses are 50 tokens and others 4k.
• High concurrency targets: you need to maximize requests per GPU to meet cost or latency SLAs.
• Long-context workloads: 16k–128k contexts where static batching’s padding waste is prohibitive.
• Prefix caching benefits: shared system prompts, few-shot examples, conversation history — PagedAttention makes prefix caching practical.
• Speculative decoding: draft and target model coordination is cleaner with block-level KV management.

Hybrid approaches

Production systems often blend both. A common pattern: continuous batching for the main serving path, with a static-batching “overflow” queue for requests that exceed a context-length threshold (where prefill would dominate an iteration). Another: disaggregated prefill — a static-batching prefill service feeds KV caches into a continuous-batching decode service. This separates the compute-bound and memory-bound phases onto different GPU pools.

Verdict

Choose static batching if you’re building offline pipelines, running evaluations, targeting hardware without PagedAttention support, or serving a narrow workload where every request looks the same. The implementation is a few hundred lines; the behavior is predictable.

Choose continuous batching for any production online serving workload with variable sequence lengths, high concurrency requirements, or long contexts. The throughput gains (2–4× typical) and memory efficiency are too large to ignore. Use vLLM, TGI, or SGLang rather than building your own — the scheduler edge cases are subtle and well-explored in those codebases.

Consider hybrid when you hit the prefill-decode interference wall at scale. Disaggregated prefill/decode is the next architectural step, not a replacement for continuous batching.