How continuous batching improves LLM throughput

Continuous batching LLM throughput is the difference between a GPU sitting idle between requests and one that stays saturated through variable load. Traditional static batching groups requests at the start of generation and blocks until all finish, but continuous batching inserts and evicts sequences as they complete. This guide walks through a concrete path to adopt and tune it in your own serving stack.

What continuous batching actually does

A transformer decoder generates tokens one step at a time. In a naive loop, you feed a single sequence through the model, get one token, repeat. The GPU math units are busy for a few milliseconds and then stall while you move data and schedule the next step.

Static batching improves this by packing N sequences into one matrix multiply. But it forces all N requests to start together and end together. If one sequence finishes early, its slots are wasted. If one is longer, everyone waits.

Continuous batching LLM throughput solves this by maintaining a global pool of active sequences. After every decoding step, finished sequences are removed and newly arrived requests are inserted into free slots. The scheduler runs a new batch immediately, so the GPU always processes the maximum number of live sequences the memory budget allows.

The core idea is not new—it is essentially iteration-level scheduling. Implementations like vLLM, TensorRT-LLM, and Hugging Face TGI use different names (continuous batching, in-flight batching) but share the same mechanism: decouple request arrival from batch boundaries.

Why static batching leaves performance on the table

Consider a workload with 100 requests: 90 are 32-token answers, 10 are 512-token answers. With static batching sized at 10 requests per batch, the first batch containing a long request forces the other 9 short ones to wait ~480 extra steps. GPU utilization collapses because the short requests could have been replaced by fresh work.

Continuous batching LLM throughput recovers those cycles. The short requests free their KV-cache slots after 32 steps; the scheduler backfills with queued requests while the long ones keep running. Measured utilization shifts from often under 20% to well above 70% on the same hardware, depending on traffic shape.

Step 1: Establish a baseline throughput measurement

You cannot tune what you do not measure. Stand up your current serving path—even if it is a single sequential loop—and record aggregate tokens per second and p99 latency under realistic concurrency.

A minimal client-side probe with the OpenAI SDK:

import asyncio, time
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="empty")

async def complete(prompt):
    resp = await client.chat.completions.create(
        model="local-model",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=128,
    )
    return resp.usage.completion_tokens

async def main():
    prompts = ["Summarize continuous batching."] * 64
    start = time.perf_counter()
    toks = await asyncio.gather(*[complete(p) for p in prompts])
    elapsed = time.perf_counter() - start
    print(f"{sum(toks)} tokens in {elapsed:.2f}s => {sum(toks)/elapsed:.1f} tok/s")

asyncio.run(main())

Run this against your existing endpoint. The number you get is your floor. Everything below assumes you will repeat this test after each change.

Step 2: Pick a serving runtime that implements it

You have three practical options if you self-host:

• vLLM: Python-centric, fast to deploy, strong continuous batching via PagedAttention.
• TensorRT-LLM: Closest to metal on NVIDIA GPUs, more complex build.
• Hugging Face TGI: Rust+Python, solid defaults, supports many model architectures.

If you would rather not operate the runtime yourself, an OpenAI-compatible gateway such as n4n.ai fronts 240+ models with provider-side continuous batching and automatic fallback when a provider is degraded, so you get the throughput without running the scheduler.

Whichever you choose, confirm the scheduler documentation explicitly mentions iteration-level or continuous batching. Some older examples labelled “dynamic batching” only batch at the prefill stage and still block on decode.

Step 3: Configure batch and token limits

Out of the box, most runtimes cap concurrent sequences conservatively. Raise the limit until you hit memory or latency walls.

For vLLM:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --max-num-seqs 256 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.92

--max-num-seqs is the ceiling on live sequences. --gpu-memory-utilization controls how much VRAM the KV-cache allocator grabs. Push utilization high, but leave headroom for fragmentation and weights.

Tradeoff: a larger batch increases throughput per watt but raises the median time-per-step. Watch p99; if it climbs past your SLA, pull the sequence cap back.

Step 4: Handle heterogeneous request shapes

Real traffic is not uniform. Continuous batching LLM throughput benefits most when short and long requests mix, but you must avoid padding penalties.

Enable chunked prefill if your runtime supports it (vLLM does). This splits long prompts into chunks that fit inside the decode batch, preventing a single 8k prompt from monopolizing the scheduler.

# vLLM flag (recent versions)
--enable-chunked-prefill

Also set a per-request max_tokens ceiling at the proxy or application layer. Unbounded generation silently consumes KV-cache and starves the batch. Reject or truncate requests asking for >2048 tokens if your product does not need them.

Step 5: Track queuing latency, not just token throughput

Throughput graphs look great right up until the queue depth grows. Continuous batching keeps the GPU busy, but if arrival rate exceeds service rate, requests wait in the scheduler.

Export the runtime’s queue metrics. In vLLM, the /metrics endpoint exposes vllm:num_requests_waiting. Alert when that gauge stays above zero for more than a few seconds.

# Prometheus query example
# avg(vllm:num_requests_waiting) > 0

If the queue never drains, you have two levers: reduce max-num-seqs to protect latency (counterintuitive but lowers step time) or scale out replicas. Do not just max out the batch size and hope.

Common pitfalls and tradeoffs

OOM from overcommitted KV-cache. The scheduler allocates per-sequence cache blocks. A misconfigured --max-num-seqs with long context will crash the worker. Test with your real context distribution, not a toy prompt.

Tail latency for long sequences. Continuous batching does not eliminate head-of-line blocking entirely. A 4k-token generation shares the batch with dozens of short ones; its step time increases slightly. If your product has strict long-output SLAs, isolate those requests on a separate pool.

Cache fragmentation. PagedAttention mitigates this, but extreme variance in sequence lengths can still leave unusable blocks. Keep --max-model-len close to your actual need.

Batching across models. Do not assume you can mix Llama-3-8B and Mixtral in the same continuous batch—they require different weights. Gateway-level routing must pin a model per replica.

Production rollout checklist

1. Baseline current tok/s and p99 with the Step 1 script.
2. Deploy vLLM or TGI with continuous batching enabled.
3. Raise --max-num-seqs in increments of 32, re-running the benchmark each time.
4. Enable chunked prefill if prompt lengths vary widely.
5. Scrape queue depth and GPU util; set alerts on both.
6. Cap max_tokens at the API boundary.
7. Keep a separate replica group for latency-sensitive long outputs.

Continuous batching LLM throughput is not magic, but it is the single highest-leverage change you can make to a serving tier short of buying more silicon. Measure, raise limits conservatively, and watch the queue.