How batch size affects LLM latency and throughput

Batch size is the single most important knob for tuning the batch size latency throughput curve in LLM inference. Every production system eventually hits the same wall: small batches give you low latency but waste GPU compute, while large batches maximize throughput but punish tail latency. Understanding why this tradeoff exists — and where the inflection points lie for your specific workload — separates systems that scale gracefully from those that require constant firefighting.

The fundamental tension

LLM inference has two distinct phases with radically different compute profiles. Prefill processes the entire prompt in parallel — all tokens at once — making it compute-bound and highly parallelizable. Decode generates tokens sequentially, one at a time, making it memory-bandwidth-bound. When you batch requests together, you’re trying to amortize the fixed costs of model loading and kernel launches across multiple sequences.

The problem: prefill and decode have opposite batching preferences. Prefill loves large batches because the matrix multiplications (batch × seq_len × hidden_dim) saturate tensor cores efficiently. Decode hates large batches because each token generation step requires reading the entire KV cache from HBM, and larger batches mean more KV cache pressure and longer memory-bound kernels.

# Simplified view of the two phases
def prefill(batch_size, seq_len, hidden_dim):
    # Batched matmul: [batch_size, seq_len, hidden_dim] @ [hidden_dim, hidden_dim]
    # Compute-bound, scales well with batch_size
    return attention(Q, K, V)  # All tokens processed in parallel

def decode(batch_size, hidden_dim, kv_cache_len):
    # Single token per sequence: [batch_size, 1, hidden_dim] @ [hidden_dim, hidden_dim]
    # Memory-bound: must read KV cache of shape [batch_size, kv_cache_len, hidden_dim]
    # Larger batch_size = more HBM reads per token
    return attention(Q, K_cache, V_cache)

This asymmetry creates the core tension: the batch size that maximizes prefill throughput destroys decode latency, and vice versa.

How continuous batching changes the equation

Traditional static batching waits for a full batch to accumulate, runs prefill on all sequences, then runs decode step-by-step until all sequences finish. This creates “bubbles” — GPU cycles wasted waiting for the slowest sequence in the batch to complete.

Continuous batching (also called iteration-level scheduling) removes completed sequences from the batch and admits new requests in their place at each decode step. This keeps the batch size near a target value continuously, dramatically improving GPU utilization.

# Continuous batching loop (simplified)
batch = []
while True:
    # Admit new requests up to target_batch_size
    while len(batch) < target_batch_size and queue:
        batch.append(queue.pop())
    
    if not batch:
        continue  # Wait for work
    
    # Single decode step for all active sequences
    logits = model.decode_step(batch)
    
    # Sample next tokens
    next_tokens = sample(logits)
    
    # Update KV caches, check for completion
    new_batch = []
    for req, token in zip(batch, next_tokens):
        req.kv_cache.append(token)
        if token == EOS or len(req.kv_cache) >= req.max_tokens:
            complete(req)
        else:
            new_batch.append(req)
    
    batch = new_batch

With continuous batching, you can run at much higher effective batch sizes without the tail-latency penalty of static batching. The batch size becomes a steady-state target rather than a maximum wait threshold.

KV cache: the hidden batch size constraint

Every active sequence in a batch carries its KV cache — two tensors per layer (keys and values) of shape [batch_size, seq_len, num_heads, head_dim]. For a 70B model with 80 layers, 8192 context length, and FP16: that’s roughly 2.6 GB per sequence. At batch size 32, you’re looking at 83 GB just for KV cache — exceeding a single H100’s 80 GB HBM.

This forces hard limits on maximum batch size. You have three escape hatches:

Quantize KV cache to FP8 or INT4 — cuts memory 2-4x with minimal quality loss
Paged attention / vLLM-style block management — allows non-contiguous KV cache allocation, reducing fragmentation and enabling larger effective batches
Multi-GPU tensor parallelism — splits KV cache across devices, but adds communication overhead

# KV cache memory calculation
def kv_cache_bytes(batch_size, seq_len, num_layers, num_heads, head_dim, dtype_bytes=2):
    # 2 tensors (K and V) per layer
    return 2 * batch_size * seq_len * num_layers * num_heads * head_dim * dtype_bytes

# 70B-ish config: 80 layers, 64 heads, 128 head_dim
# At batch=32, seq_len=4096, FP16:
print(kv_cache_bytes(32, 4096, 80, 64, 128, 2) / 1e9)  # ~41.5 GB
# At batch=32, seq_len=8192: ~83 GB — exceeds single H100

The practical batch size ceiling is often set by KV cache memory, not compute. If you’re not using paged attention and KV quantization, you’re leaving 2-4x throughput on the table.

Latency vs throughput curves in practice

The relationship isn’t linear. Here’s what you typically observe on a single H100 running a 70B model with continuous batching and FP8 KV cache:

Batch size	Prefill throughput (tok/s)	Decode throughput (tok/s)	P50 latency (ms/tok)	P99 latency (ms/tok)
1	~800	~120	8	15
4	~2,800	~400	10	22
16	~5,200	~1,100	15	45
32	~6,000	~1,600	22	95
64	~6,300	~1,800	38	280

Notice the diminishing returns: batch 16 to 32 gives ~45% more decode throughput but 2.5x worse P99 latency. Batch 32 to 64 gives only ~12% more throughput but 3x worse P99. The knee of the curve typically sits around batch 16-32 for decode on a single GPU.

Prefill scales more linearly because it’s compute-bound, but you rarely run pure prefill workloads — real traffic is a mix.

The request-size distribution matters

Average batch size is a misleading metric if your request sizes vary wildly. A batch of 32 sequences at 128 tokens each behaves very differently from 32 sequences at 8,192 tokens. The latter consumes 64x more KV cache memory and keeps sequences in the batch 64x longer, blocking new admissions.

Production systems need admission control that accounts for estimated KV cache footprint, not just sequence count:

class AdmissionController:
    def __init__(self, max_kv_blocks, block_size=16):
        self.max_blocks = max_kv_blocks
        self.block_size = block_size
        self.used_blocks = 0
    
    def estimate_blocks(self, prompt_tokens, max_new_tokens):
        total_tokens = prompt_tokens + max_new_tokens
        return (total_tokens + self.block_size - 1) // self.block_size
    
    def can_admit(self, prompt_tokens, max_new_tokens):
        needed = self.estimate_blocks(prompt_tokens, max_new_tokens)
        return self.used_blocks + needed <= self.max_blocks
    
    def admit(self, prompt_tokens, max_new_tokens):
        needed = self.estimate_blocks(prompt_tokens, max_new_tokens)
        self.used_blocks += needed
        return True
    
    def release(self, prompt_tokens, max_new_tokens):
        needed = self.estimate_blocks(prompt_tokens, max_new_tokens)
        self.used_blocks -= needed

This prevents a few long-context requests from starving the system of capacity for short requests.

When to prefer latency over throughput (and vice versa)

Optimize for latency (batch 1-8) when:

Real-time user-facing chat with streaming — users perceive per-token latency directly
Interactive coding assistants — sub-50ms/token feels instant; 100ms+ feels laggy
Low-traffic internal tools — GPU utilization matters less than responsiveness
Speculative decoding workloads — small batches keep the draft model’s acceptance rate high

Optimize for throughput (batch 16-64+) when:

Batch/async workloads — document processing, embedding generation, offline summarization
High-traffic APIs where cost-per-token dominates — you’re paying for GPU hours
Providers serving many concurrent users — continuous batching smooths the latency distribution
Workloads with predictable, uniform request sizes — easier to tune a stable operating point

The hybrid approach: Run two separate model replicas (or two pools in a disaggregated architecture). Route latency-sensitive traffic to a low-batch-size pool, throughput traffic to a high-batch-size pool. This avoids the “one batch size fits all” compromise.

Disaggregated prefill and decode

The logical next step is splitting prefill and decode onto separate GPU pools. Prefill GPUs run large batches (64-128+) for maximum throughput on the compute-bound phase. Decode GPUs run smaller batches (8-16) for latency. Sequences move from prefill to decode after the first token.

This architecture — sometimes called “disaggregated inference” or “prefill-decode separation” — lets you tune each phase independently. The tradeoff: you need a fast KV cache transfer mechanism (NVLink, RDMA, or shared memory) and a scheduler that balances load across pools.

# Conceptual disaggregated flow
class DisaggregatedScheduler:
    def __init__(self, prefill_pool, decode_pool):
        self.prefill_pool = prefill_pool  # Large batch, high throughput
        self.decode_pool = decode_pool    # Small batch, low latency
    
    async def handle_request(self, request):
        # Phase 1: Prefill on large-batch pool
        kv_cache = await self.prefill_pool.prefill(request.prompt)
        
        # Phase 2: Transfer KV cache to decode pool
        # (In practice: zero-copy via NVLink or shared memory)
        decode_handle = await self.decode_pool.admit(kv_cache, request.max_tokens)
        
        # Phase 3: Stream decode tokens
        async for token in self.decode_pool.stream(decode_handle):
            yield token

This is where systems like n4n.ai’s routing layer add value — directing prefill-heavy and decode-heavy workloads to appropriately configured model instances without client-side complexity.

Practical tuning checklist

If you’re deploying today, start here:

Enable continuous batching — vLLM, TGI, TensorRT-LLM all support this. It’s table stakes.
Quantize KV cache to FP8 — 2x memory savings, negligible quality impact on most models.
Use paged attention — Eliminates fragmentation, lets you run larger effective batches.
Set a target batch size, not a max — Continuous batching works best with a steady-state target (e.g., 16-32 for decode).
Monitor P99 decode latency per token — Not just throughput. If P99 > 100ms/token, your batch size is too high for interactive traffic.
Implement KV-cache-aware admission — Block long requests from starving short ones.
Benchmark your specific model + hardware — The curves above are illustrative. Your knee point depends on model size, quantization, GPU generation, and context length distribution.

The decisive takeaway

Batch size is not a tuning parameter you set once. It’s a runtime control variable that should adapt to your traffic mix. For interactive workloads, target the smallest batch size that keeps your GPUs above 70% compute utilization during decode — typically 8-16 on modern hardware with continuous batching. For throughput workloads, push to the KV cache memory limit (32-64+ with FP8 + paged attention).

The systems that win don’t pick a batch size. They build admission control, scheduling, and routing that dynamically operate near the knee of the latency-throughput curve for each workload type. Everything else is just guessing.