Continuous batching explained: how it beats static batching

Continuous batching explained simply: it lets you add new requests to a batch the moment a slot frees up, instead of waiting for every sequence in the batch to finish. Static batching pads all sequences to the same length and processes them in lockstep, wasting compute on padding tokens and idling while the longest request completes. The difference shows up immediately in throughput, tail latency, and GPU utilization — especially under variable-length workloads.

What static batching does

Static batching groups incoming requests into fixed-size batches, pads every sequence to the maximum length in that batch, and runs the entire batch through the model together. The scheduler waits until all sequences hit their stop condition before releasing the batch and pulling the next one.

# Static batching pseudocode
def static_batch_step(requests, max_batch_size=32):
    batch = requests[:max_batch_size]
    max_len = max(len(r.input_ids) for r in batch)
    padded = [pad(r.input_ids, max_len) for r in batch]
    logits = model.forward(padded)  # shape: [batch, max_len, vocab]
    return [logits[i, len(batch[i].input_ids)-1] for i in range(len(batch))]

This approach is simple to implement and works fine when request lengths are uniform — think classification, embedding, or short chat turns with similar token counts. It breaks down when you mix 50-token prompts with 2,000-token prompts. The short requests sit idle while the long one finishes, and you burn FLOPs on padding tokens that contribute nothing to the output.

What continuous batching does

Continuous batching (also called iteration-level scheduling) treats each decoding step as a scheduling opportunity. When a sequence finishes — hits EOS, max tokens, or a stop string — its slot opens immediately. The scheduler pulls the next waiting request into that slot for the very next forward pass. No padding to the batch maximum, no waiting for stragglers.

# Continuous batching pseudocode
class ContinuousBatch:
    def __init__(self, max_tokens=4096):
        self.running = []      # (request_id, input_ids, kv_cache)
        self.waiting = deque() # pending requests
        self.max_tokens = max_tokens

    def step(self):
        # 1. Forward pass on all running sequences (one token each)
        logits = self.model.forward([r.input_ids[-1:] for r in self.running])

        # 2. Sample next token, update KV caches
        new_running = []
        for (req_id, ids, kv), logit in zip(self.running, logits):
            next_token = sample(logit)
            ids.append(next_token)
            kv.append(compute_kv(next_token))
            if not is_done(next_token, ids):
                new_running.append((req_id, ids, kv))
            else:
                self.finish(req_id, ids)

        # 3. Backfill freed slots from waiting queue
        while len(new_running) < self.max_tokens and self.waiting:
            req = self.waiting.popleft()
            new_running.append((req.id, req.input_ids, init_kv(req.input_ids)))

        self.running = new_running

The key insight: each forward pass processes exactly one token per active sequence. The batch size fluctuates naturally. A 50-token request finishes in 50 steps and frees its slot; a 2,000-token request occupies its slot for 2,000 steps. Short requests never wait for long ones.

Head-to-head comparison

Latency and throughput deep dive

Static batching gives you predictable per-request latency if all requests are similar. The moment you introduce variance, the average latency stays acceptable but the tail explodes. A batch with one 4k-token request and thirty-one 100-token requests forces the short requests to wait for ~40× their natural decode time.

Continuous batching flattens the tail. Short requests finish in ~100 steps regardless of what else is running. The tradeoff: time-to-first-token becomes variable because a request may queue behind a prefill-heavy batch. Most production systems solve this by separating prefill and decode — running prefill as large static batches (high arithmetic intensity) and decode as continuous batches (memory-bound). This two-phase design is what vLLM, TGI, and TensorRT-LLM all converge on.

Throughput numbers tell the same story. On an H100 serving Llama-3-70B with a realistic mix of 200–3000 token prompts, continuous batching typically delivers 2–4× the request throughput of static batching at the same p99 latency target. The gap widens as length variance increases. If your workload is 90% similar-length requests, the gap narrows to 10–20%.

Memory and scheduling implications

Continuous batching demands a different memory allocator. Static batching allocates one contiguous KV cache per request at its maximum length. Continuous batching needs a pool of KV blocks (typically 16–64 tokens each) that can be chained, split, and freed per iteration. This enables prefix caching — if two requests share a system prompt, they reference the same physical blocks — and makes preemption feasible: evict a low-priority request’s blocks, admit a high-priority one, and resume the evicted request later by recomputing its prefix.

The scheduler also gets richer. You can implement:

• Priority lanes: dedicate a fraction of decode slots to high-priority traffic
• Speculative decoding: run a small draft model ahead, verify with the target model in the same continuous batch
• Chunked prefill: split long prefill across multiple iterations to avoid blocking decode slots

None of this is free. The kernel complexity is real: you need custom attention kernels that handle ragged sequences (variable lengths per request in a single batch), block-table lookups for KV cache, and efficient prefix-sharing logic. This is why most teams adopt vLLM, SGLang, or TensorRT-LLM rather than building from scratch.

Which to choose

Choose static batching when:

• Your requests are uniform in length (embeddings, classification, reranking, short QA with tight length bounds)
• You need the simplest possible implementation and can tolerate lower GPU utilization
• You’re running offline/batch inference where latency doesn’t matter and you can sort requests by length to minimize padding

Choose continuous batching when:

• You serve open-ended generation (chat, code, creative writing) where prompt and output lengths vary widely
• Tail latency matters — p99 targets, SLOs, user-facing applications
• You need prefix caching for shared system prompts or few-shot examples
• You want to run speculative decoding or priority scheduling
• GPU utilization and cost-per-token are primary concerns

Hybrid approach (what production systems actually do): Run prefill as large static batches (maximize tensor core utilization on the compute-heavy prompt phase) and decode as continuous batches (maximize memory bandwidth utilization on the memory-bound token-by-token phase). Route requests through a prefill queue and a decode queue with separate scheduler policies. This is the architecture behind vLLM, TGI, and the inference stack at n4n.ai — one OpenAI-compatible endpoint that handles 240+ models with automatic fallback when a provider degrades, per-token usage metering, and client routing directives that let you pin traffic to specific backends.

If you’re building a serving stack today, start with continuous batching for decode. The engineering investment pays off the moment your first variable-length workload hits production.