The relationship between batch size tokens per second is the primary determinant of GPU cost efficiency once you move past a single user. Most teams tune for first-token latency on isolated requests, then deploy and watch utilization crater because the scheduler never packs sequences. This guide lays out an ordered path to measure, batch, and scale inference throughput without sacrificing the tail latency your users feel.
1. Measure baseline throughput before touching batch size
You cannot improve what you have not measured. Stand up a single synchronous request against your model and record time-to-first-token (TTFT), tokens generated, and wall-clock duration.
import time, openai
client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
start = time.perf_counter()
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain batching in 100 words."}],
stream=True,
)
tokens = 0
for chunk in stream:
if chunk.choices[0].delta.content:
tokens += 1
elapsed = time.perf_counter() - start
print(f"Generated ~{tokens} tokens in {elapsed:.2f}s -> {tokens/elapsed:.1f} tok/s")That number is your single-stream tokens per second. The gap between it and aggregate cluster output is where batch size tokens per second gains live.
2. Separate throughput from latency
Batch size tokens per second is a throughput metric: total generated tokens across all concurrent requests divided by elapsed time. Latency is what a single user experiences: TTFT and inter-token gap.
Increasing batch size improves throughput by amortizing attention and matmul overhead across sequences. But it increases TTFT because the scheduler must wait to assemble a batch before the first forward pass. Past a point, you are trading user-perceived responsiveness for cheaper tokens.
Common pitfall: optimizing mean tokens per second while p99 TTFT climbs past 2 seconds. Define a latency SLO first (e.g., TTFT < 400 ms at p50, < 1 s at p99) and treat batch size as the dial that fills the GPU until that SLO bends. The batch size tokens per second curve only matters inside that envelope.
3. Use continuous batching, not static padding
Static batching groups N requests, pads them to the longest sequence, and runs one big matmul. Padding wastes memory and compute on nonsense tokens. Continuous batching (implemented in vLLM, TensorRT-LLM, and others) appends finished requests and admits new ones at each decoding step.
If you run your own serving stack, enable it:
{
"engine": "vllm",
"tensor_parallel_size": 2,
"enable_continuous_batching": true,
"max_num_seqs": 32,
"max_model_len": 8192
}The max_num_seqs parameter is your hard batch ceiling. Set it from memory math, not guesswork (next section). Continuous batching makes the effective batch size tokens per second curve far smoother because the GPU is rarely starved or overloaded.
4. Size batches against KV cache, not just VRAM
The limiting factor for batch size is almost always KV cache memory, not model weights. Weights are fixed; KV cache grows with sequence length and concurrency. For grouped-query attention, account for reduced KV heads.
Estimate bytes per token per sequence:
kv_bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytesFor a 7B model with 32 layers, 32 KV heads, head_dim 128, fp16 (2 bytes):
num_layers=32; num_kv_heads=32; head_dim=128; dtype_bytes=2
per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytes
print(per_token) # 524288 bytes ~ 0.5 MB per token per sequenceIf you have 16 GB free for KV after weights, and average context is 1024 tokens, max sequences ≈ 16e9 / (0.5e6 * 1024) ≈ 31. That is your realistic max batch size. Exceed it and you OOM or thrash to CPU.
Tradeoff: longer context shrinks max batch dramatically. A 32k context cuts concurrency 32x. Design batch size tokens per second expectations around your actual prompt distribution, not benchmark max. If you serve mixed lengths, split into pools: one for long-context, one for short.
5. Drive concurrency from a server-side queue
Do not let clients fire arbitrary parallel requests and hope the scheduler copes. Put a bounded queue in front of the model and admit only as many sequences as max_num_seqs allows.
import asyncio, openai
sem = asyncio.Semaphore(32) # matches max_num_seqs
client = openai.AsyncOpenAI()
async def gen(prompt):
async with sem:
resp = await client.chat.completions.create(
model="gpt-4o-mini", messages=[{"role":"user","content":prompt}], stream=True)
async for _ in resp:
passThis caps batch size tokens per second at the hardware limit and prevents latency blowups from overload. Clients get 429s when full; handle with backoff. A queue also lets you prioritize interactive traffic over background summarization.
6. Load test with production-shaped traffic
Synthetic load tests using 8-token prompts will report absurd tokens per second and lead you to oversize batches. Record real prompt lengths and output lengths, then replay.
A minimal locustfile:
from locust import User, task, between
import openai
class LLMUser(User):
wait_time = between(1, 5)
@task
def complete(self):
openai.OpenAI().chat.completions.create(
model="gpt-4o-mini",
messages=[{"role":"user","content":"Summarize: " + "x"*512}],
max_tokens=256)Run with escalating user counts. Plot aggregate tokens per second vs concurrency. You will see linear growth, then a knee, then plateau. That knee is your operating batch size.
Pitfall: ignoring prefix caching. If many prompts share a system prefix, provider cache hits reduce compute per token and shift the curve upward. Forward cache-control hints from clients when your gateway supports it. Another pitfall: measuring only mean. Watch p95 tokens per second; a few stalled sequences can drag the average while the batch is healthy.
7. Monitor per-token cost and route around degradation
Throughput at scale is meaningless if a provider goes into rate-limit or degraded mode and your batch pipeline stalls. Per-token metering lets you compare effective cost across models and providers. If you use a gateway such as n4n.ai, its automatic fallback switches to a healthy provider when one is rate-limited, preserving your batch size tokens per second instead of draining the queue.
Set up alerts on:
- Queue depth (should be near zero at steady state)
- p99 TTFT (SLO breach)
- Tokens per second per GPU (utilization)
Tradeoff: fallback may route to a slower model, lowering per-batch throughput. Accept it; a slower batch beats a failed one. Honor client routing directives when you need to pin a batch to a specific provider with spare KV capacity.
8. Autoscale on queue depth, not GPU utilization
Horizontal scaling should trigger when queue depth exceeds max_num_seqs * k for sustained periods, not when VRAM hits 80%. VRAM stays high even when the batch is small if contexts are long.
# example prometheus alert
alert: BatchQueueBacklog
expr: llm_queue_depth > 64
for: 2mAdd replicas with the same max_num_seqs config. Because continuous batching absorbs variance, you rarely need more than 2-3 replicas per traffic tier if sized correctly. Over-provisioning replicas wastes the very GPU budget you were trying to optimize.
9. Common pitfalls and tradeoffs summary
- Over-batching: TTFT violates SLO; users perceive lag despite great tokens per second.
- Under-batching: GPU sits idle; cost per token skyrockets.
- Ignoring KV fragmentation: continuous batching helps but memory fragmentation still caps real concurrency.
- Mixing long and short contexts: one 32k request can evict dozens of short ones; use separate pools.
- Treating batch size tokens per second as a single number: it is a curve dependent on sequence length distribution.
Tune the curve, not the peak. Measure with real traffic, cap concurrency at the KV cache limit, and scale horizontally only when the queue proves the local GPU is saturated. That is how you turn idle GPUs into predictable, cheap token factories.