Batch size tuning for maximum LLM serving throughput

Batch size tuning throughput is the highest-leverage knob for squeezing more tokens per dollar from an LLM serving stack. Most teams pick a batch size once during initial deploy and never revisit it, leaving 2–4x of potential GPU utilization on the table. This guide walks through a measurement-driven process to find the optimal batch size for your model, hardware, and traffic shape, with runnable code at each stage.

Step 1: Establish a single-request baseline

Before sweeping anything, measure what one request costs in time and tokens. You need the latency and token rate of an isolated call to know how much headroom exists. Server-side caches, CUDA graph warmup, and framework lazy loading all distort the first call, so always discard the initial request.

import time, openai

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
# warmup
client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "hi"}],
    max_tokens=8,
)
t0 = time.time()
resp = client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Explain batch sizing in 100 words."}],
    max_tokens=128,
)
dt = time.time() - t0
print(f"latency={dt:.2f}s tokens/s={resp.usage.completion_tokens/dt:.1f}")

Run this three times and take the median. If your single-stream token rate is far below the hardware spec (e.g., under 30% of published tokens/s for the GPU class), the server is likely under-batched or misconfigured before you even start. Note the number; it is your denominator for speedup calculations.

Step 2: Define your workload shape

Throughput optimization is meaningless without a representative request distribution. Capture two numbers from production logs or synthetic generators:

• Prompt length (input tokens): mean and p99.
• Generation length (output tokens): mean and max.

A chatbot with 512-token prompts and 128-token answers behaves differently from a summarizer with 4K-token inputs and 256-token outputs. Batch size tuning throughput depends heavily on these because KV cache memory scales with (prompt + generation) length per sequence, not just request count. A batch of 32 short requests may fit where a batch of 8 long ones overflows.

Write a small generator that emits requests matching those lengths. Use your actual tokenizer in production; for benchmarks, repeated words approximate token counts closely enough.

def make_messages(prompt_tok=512, gen_tok=128):
    # approximate with repeated text; real systems use tokenizer
    return [{"role": "user", "content": "word " * prompt_tok}], gen_tok

Keep the generation length fixed during the sweep. Variable lengths complicate throughput math because the batch finishes only when the longest sequence completes unless you use continuous batching (covered later).

Step 3: Sweep batch sizes systematically

Increase concurrency in powers of two and measure aggregate token throughput. Use an async client to fire N requests simultaneously and divide total completion tokens by wall-clock time. This is a closed-loop test: you are measuring saturation, not live traffic.

import asyncio, os
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

async def one_req(prompt_tok, gen_tok):
    msgs, _ = make_messages(prompt_tok, gen_tok)
    resp = await client.chat.completions.create(
        model="meta-llama/Llama-3-8B-Instruct",
        messages=msgs,
        max_tokens=gen_tok,
    )
    return resp.usage.completion_tokens

async def run_batch(n, prompt_tok=512, gen_tok=128):
    t0 = time.time()
    tasks = [one_req(prompt_tok, gen_tok) for _ in range(n)]
    toks = await asyncio.gather(*tasks)
    dt = time.time() - t0
    return sum(toks) / dt

for bs in [1, 2, 4, 8, 16, 32, 64]:
    tput = asyncio.run(run_batch(bs))
    print(f"batch={bs} throughput={tput:.1f} tok/s")

If you route through n4n.ai, its per-token usage metering removes the need to instrument token counting yourself—each batch response includes exact usage. The curve will rise steeply, then flatten. The flattening point is your throughput knee. Stop the sweep when throughput gains drop below 5% per doubling; beyond that you are just adding latency.

Step 4: Watch the memory ceiling and decode latency

Larger batches consume more KV cache. The memory per sequence is roughly 2 * num_layers * hidden_dim * seq_len * bytes_per_elem. When the allocator hits limits, the server either rejects requests or spills to host memory, collapsing throughput. Monitor GPU memory during the sweep:

watch -n 1 nvidia-smi --query-gpu=memory.used --format=csv

At the same time, record p99 decode latency. A batch size that doubles throughput but triples time-to-first-token is a poor trade for interactive apps. For offline jobs, latency matters less; push the batch until OOM or scheduler thrash.

If you see throughput drop after a certain batch size, you have exceeded the effective context memory budget. Step back to the last stable point. Also watch for non-linear latency jumps—those signal that the scheduler is preempting sequences, which wastes compute.

Step 5: Tune the scheduler and continuation batching

Static batching wastes cycles while waiting for the slowest request in a batch to finish generating. Enable continuous (in-flight) batching in your serving framework (vLLM, TensorRT-LLM, SGLang). This lets new requests join as earlier ones complete, keeping the GPU fed and effectively raising the batch size tuning throughput ceiling without increasing peak VRAM.

Also set prefix caching if your prompts share system prefixes. Cache hits reduce prompt processing cost, letting you admit more sequences into the same memory envelope.

{
  "enable_prefix_caching": true,
  "max_num_seqs": 32,
  "max_num_batched_tokens": 4096
}

Adjust max_num_batched_tokens (total tokens in flight) alongside raw request count. This parameter often matters more than concurrent request count because it bounds memory directly. A batch of 16 requests with 4K context each needs far more headroom than 32 requests with 512-token contexts.

If your framework supports chunked prefill, turn it on. It splits long prompts into smaller chunks that interleave with decode steps, smoothing the latency curve at large batch sizes.

Step 6: Validate under production-like load

A synchronized sweep is not real traffic. Use a Poisson arrival generator or a load tool to send requests at a target QPS with jitter. This open-loop test reveals whether your tuned batch size survives bursty arrivals.

import asyncio, random, time
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

async def poisson_worker(qps, duration):
    end = time.time() + duration
    while time.time() < end:
        asyncio.create_task(one_req(512, 128))
        await asyncio.sleep(random.expovariate(qps))

asyncio.run(poisson_worker(qps=20, duration=60))

Run this against the chosen batch configuration. Observe tail latency and error rate. If the queue depth grows unbounded, your batch size is too small for the arrival rate, or the scheduler is starving. Increase max_num_seqs or add replica workers rather than inflating batch size past the knee.

Verify success

You have tuned correctly when three conditions hold:

1. Throughput plateau: Increasing batch size by 2x yields <10% token/s gain.
2. Memory stable: nvidia-smi shows <95% VRAM at p99 load, with no OOM events in logs.
3. Latency acceptable: p99 time-to-first-token meets your product SLO (e.g., <500ms for chat, <2s for async summarization).

Automate the Step 3 sweep in CI against a fixed model and hardware snapshot. Batch size tuning throughput should be a recurring check, not a one-off. When you change models, GPU types, or prompt lengths, rerun the sweep—the knee moves with every variable.

If you front the endpoint with a gateway that honors client routing directives, pin the test traffic to a single provider during benchmarks to avoid mixing hardware in your numbers. That isolates the variable you are actually measuring, and prevents a silent fallback from masking a regression.

Finally, document the winning batch size and the exact workload shape that produced it. Six months from now, a teammate will change the system prompt length and wonder why p99 latency slipped. Your benchmark notes are the only defense.