Cut inference costs with vLLM continuous batching

vLLM continuous batching reduce inference cost by keeping GPUs saturated across variable-length requests instead of padding every sequence to the same length. Traditional static batching forces you to choose between wasting compute on padding tokens or serializing requests and leaving GPU cycles idle. Continuous batching solves this by dynamically scheduling new requests into freed slots as sequences finish, turning tail latency into throughput. This guide walks through enabling it, tuning the scheduler for your workload, and measuring the actual impact.

What continuous batching actually does

Static batching groups requests by padded length. If you batch a 100-token prompt with a 2000-token prompt, the shorter request sits idle for 1900 tokens of generation. Continuous batching treats each request as an independent sequence with its own lifecycle. When a sequence hits EOS or max_tokens, its KV cache slots are immediately reclaimed and a waiting request fills the gap — all within the same forward pass.

The vLLM scheduler maintains a waiting queue and a running queue. Each iteration, it:

1. Appends new tokens to running sequences
2. Checks for finished sequences and frees their blocks
3. Pulls waiting requests into the newly freed blocks up to max_num_batched_tokens and max_num_seqs
4. Executes a single forward pass over the combined batch

This happens at the C++/CUDA level in the LLMEngine loop. You don’t manage the queues yourself; you configure the limits and the engine handles the rest.

When it helps (and when it doesn’t)

Continuous batching shines when:

• Request lengths vary significantly (chat, RAG, code generation)
• You have steady request volume — bursty traffic with long idle gaps still leaves GPUs underutilized
• Your bottleneck is compute, not memory bandwidth or KV cache capacity

It hurts or adds complexity when:

• All requests are nearly identical length (batch inference over a fixed dataset)
• You need strict per-request latency SLAs — continuous batching optimizes throughput, not tail latency
• KV cache memory pressure forces frequent eviction, adding overhead

If your p99 latency matters more than cost per token, static batching with careful padding buckets may still win.

Setting up vLLM with continuous batching

vLLM enables continuous batching by default when you use the OpenAI-compatible server. The key flags control the scheduling window:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching

--max-num-batched-tokens is the total token budget per forward pass (prompt + generated tokens across all sequences). --max-num-seqs caps concurrent sequences. Both must be set together — the scheduler respects whichever limit hits first.

For programmatic use, the same parameters apply to LLMEngine:

from vllm import LLMEngine, SamplingParams
from vllm.engine.arg_utils import EngineArgs

engine_args = EngineArgs(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    max_num_batched_tokens=8192,
    max_num_seqs=256,
    max_model_len=8192,
    gpu_memory_utilization=0.90,
    enable_prefix_caching=True,
)
engine = LLMEngine.from_engine_args(engine_args)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
request_id = "req-001"
engine.add_request(request_id, "Explain continuous batching in two sentences.", sampling_params)

while engine.has_unfinished_requests():
    outputs = engine.step()
    for output in outputs:
        if output.finished:
            print(f"{output.request_id}: {output.outputs[0].text}")

The step() call runs one scheduler iteration. In production you’d wrap this in an async server; the OpenAI-compatible endpoint does this for you.

Tuning the scheduler for your workload

Start with max_num_batched_tokens ≈ max_model_len × 1.5. For Llama-3-8B with 8K context, 12K–16K is a reasonable starting point. Monitor two metrics:

• Batch size per step (sequences actually running)
• Token budget utilization (tokens used / max_num_batched_tokens)

If token budget utilization sits below 60% consistently, raise max_num_batched_tokens. If batch size hits max_num_seqs but token budget has headroom, raise max_num_seqs. The sweet spot is both limits binding simultaneously.

Prefix caching (--enable-prefix-caching) multiplies the effect. Shared system prompts, few-shot examples, or RAG contexts reuse KV blocks across requests, effectively increasing the token budget without more GPU memory. Enable it unless your workload has zero prefix overlap.

# Verify prefix caching is active
from vllm import LLMEngine
engine = LLMEngine.from_engine_args(EngineArgs(enable_prefix_caching=True))
# Check block manager type
print(type(engine.scheduler.block_manager).__name__)
# Should show 'PrefixCachingBlockManager'

Chunked prefill for long contexts

When prompts exceed ~4K tokens, prefill dominates step time and starves decode. Chunked prefill splits long prompts across multiple forward passes, interleaving decode steps. Enable with:

--enable-chunked-prefill --max-num-batched-tokens 16384

This adds a prefill_chunk_size parameter (default 8192). Tune it so prefill chunks fit in the token budget alongside active decode sequences. For 8K context models, 4K–8K chunk size works well.

Measuring the difference

Instrument your server to emit per-step metrics. vLLM logs scheduler stats at INFO level; capture them or expose via Prometheus:

import logging
logging.getLogger("vllm.scheduler").setLevel(logging.INFO)

Key log fields:

• num_running — sequences in decode
• num_waiting — queued requests
• token_budget_used / token_budget_total
• prefill_tokens / decode_tokens per step

Run a load test with realistic length distribution. A simple Locust script:

from locust import HttpUser, task, between
import random

class VLLMUser(HttpUser):
    wait_time = between(0.1, 0.5)
    
    @task
    def chat_completion(self):
        prompt_len = random.choice([100, 500, 2000, 5000])
        prompt = "x " * prompt_len  # placeholder; use real prompts in practice
        self.client.post(
            "/v1/chat/completions",
            json={
                "model": "meta-llama/Meta-Llama-3-8B-Instruct",
                "messages": [{"role": "user", "content": prompt}],
                "max_tokens": 512,
                "temperature": 0.7,
            },
        )

Run for 10 minutes at target QPS. Compare:

• Tokens/second/GPU (throughput)
• Cost per 1M tokens (throughput × GPU hourly cost)
• p50/p99 latency (time to first token, time per output token)

Typical gains: 1.5–3× throughput over static batching at same latency, or 30–50% lower latency at same throughput. Your exact numbers depend on length variance and prefix overlap.

Common pitfalls

OOM from aggressive token budgets

Setting max_num_batched_tokens too high causes CUDA OOM during prefill spikes. The scheduler checks budget before allocating blocks, but prefill chunks can briefly exceed it. Leave 10–15% headroom on gpu_memory_utilization and monitor torch.cuda.max_memory_allocated().

Starving long requests

If short requests flood the queue, long requests wait indefinitely. vLLM uses FCFS within the waiting queue. For fairness, implement priority lanes at the application layer or use --scheduling-policy priority (experimental) with explicit priority per request.

Ignoring KV cache fragmentation

Continuous batching creates variable-sized holes in the block table. The block manager compacts automatically, but under extreme churn (many short requests), fragmentation can reduce usable blocks by 10–20%. Monitor block_manager.num_free_blocks vs num_total_blocks.

Disabling prefix caching by accident

Prefix caching requires enable_prefix_caching=True and deterministic tokenization for shared prefixes. If you prepend random request IDs or timestamps to every prompt, you defeat it. Keep system prompts identical across requests.

Tradeoffs to consider

If you serve a chat product with variable context, continuous batching is the default choice. If you run nightly batch jobs over fixed-length documents, static batching with max_num_seqs=1 and large max_num_batched_tokens may be simpler and more predictable.

Putting it in production

Deploy behind a load balancer with multiple vLLM replicas. Each replica runs continuous batching independently. Route requests with consistent prefixes (same system prompt, same RAG corpus) to the same replica to maximize prefix cache hits — a simple consistent hash on the system prompt hash works.

upstream vllm_backend {
    ip_hash;  # or hash $system_prompt_hash consistent;
    server gpu-1:8000;
    server gpu-2:8000;
    server gpu-3:8000;
}

Set max_num_seqs per replica based on GPU memory. For H100 80GB with Llama-3-8B, 256–512 sequences is typical. Scale replicas horizontally for QPS; scale max_num_seqs vertically for burst absorption.

Monitor the scheduler queue depth. If num_waiting grows unbounded, you’re under-provisioned. Add replicas or shed load with 429 responses before queue latency dominates.

Continuous batching is the single highest-leverage knob for vLLM inference cost. Enable it, tune the two budget parameters against real traffic, and measure tokens per dollar. The rest is operational hygiene.