Every inference service lives on a curve where the throughput latency batch size tradeoff dictates how many requests you can serve per dollar versus how fast each answer returns. Push batch size up and you amortize fixed prefill overhead across more tokens; push it down and you minimize time-to-first-token for interactive users. This guide gives an ordered path to find the operating point that fits your latency budget without guessing.
The core tradeoff
LLM inference splits into prefill (process prompt tokens) and decode (generate token by token). Prefill is compute-bound and benefits from large matrix multiplies; decode is memory-bandwidth-bound and runs one token per step. When you batch requests, you merge their prefills into a single larger matmul and share GPU kernels across sequences during decode.
The throughput latency batch size tradeoff emerges because bigger batches raise arithmetic intensity and GPU utilization, increasing tokens/sec. But each decode step must iterate over all sequences in the batch, so per-token latency grows with batch size. At some point KV cache memory or bandwidth saturates, and latency climbs faster than throughput improves.
Key metrics:
- Throughput: output tokens/sec or completed requests/sec.
- Latency: p50/p95 request-to-response time, including queue wait.
- Batch size: concurrent sequences processed together.
Measure before tuning
Never tune blind. Stand up a measurement harness using an OpenAI-compatible client. When running sweeps against an OpenAI-compatible gateway such as n4n.ai, per-token usage metering lets you pair latency measurements with exact cost per batch size, which exposes the true economy of larger batches.
from openai import OpenAI
import time, statistics
client = OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
def single_latency(prompt, max_tokens=128):
start = time.perf_counter()
client.chat.completions.create(
model="mistral-7b",
messages=[{"role":"user","content":prompt}],
max_tokens=max_tokens,
)
return time.perf_counter() - startWarm up the endpoint, then collect 100 samples for a fixed prompt. Use a prompt that matches production shape; if your traffic has shared prefixes, be aware that some gateways forward provider cache-control hints that will skew repeated calls.
samples = [single_latency("Explain the batch size tradeoff") for _ in range(100)]
p50 = statistics.median(samples)
p95 = sorted(samples)[int(0.95 * len(samples))]
print(f"p50={p50*1000:.1f}ms p95={p95*1000:.1f}ms")Step 1: Set your latency budget
Define the SLO before touching batch size. Interactive chat typically targets p50 < 300ms and p95 < 800ms. Background extraction may tolerate p95 < 5s. This budget caps how long you can queue requests before flushing a batch.
Write it down:
- Max acceptable p95:
__ms - Target throughput:
__ req/s - Cost ceiling:
__ $/1M tokens
Step 2: Profile single-stream baseline
Run sequential requests to get the floor latency with zero queue contention. This is your S_base. Any batching adds queue wait W and compute inflation k (where k ≥ 1 multiplies decode time). Total latency ≈ W + k * S_base.
If S_base p95 is already near your budget, batching will not fit interactive use; you need model or hardware changes.
Step 3: Sweep batch sizes
Most gateways accept concurrent requests; true server-side batching may require a specific endpoint. Simulate concurrency with async clients.
import asyncio
from openai import AsyncOpenAI
aclient = AsyncOpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
async def run_batch(batch_size, prompt, max_tokens=128):
tasks = [aclient.chat.completions.create(
model="mistral-7b",
messages=[{"role":"user","content":prompt}],
max_tokens=max_tokens) for _ in range(batch_size)]
start = time.perf_counter()
await asyncio.gather(*tasks)
return time.perf_counter() - start
for bs in [1, 2, 4, 8, 16, 32, 64]:
dur = asyncio.run(run_batch(bs, "Summarize the tradeoff"))
print(f"bs={bs:2d} total={dur*1000:.0f}ms throughput={bs/dur:.1f} req/s")Record both total latency and throughput. Repeat each size three times to absorb variance.
Step 4: Locate the knee
Plot throughput (y) vs batch size (x). Early growth is steep; later it flattens. Simultaneously plot p95 latency (y2) – it rises sublinearly then often explodes.
The knee is where:
- Throughput gain from
bstobs*2is < 15% - p95 latency increase exceeds your SLO headroom
Operating at the knee gives most of the throughput win at a fraction of the latency cost. Past the knee you burn memory for little benefit.
Step 5: Model queueing effects
Batching introduces a queue. If arrival rate is λ requests/sec and you flush every W seconds with up to B requests, effective service time per batch is S_batch ≈ S_base * f(B). Average latency per request ≈ W/2 + S_batch.
Using M/M/1 intuition: as utilization ρ approaches 1, latency diverges. Batching reduces ρ by raising effective capacity, but only if W is small. A common pitfall is setting W too high (e.g., 200ms) to fill large batches, which pushes p50 beyond interactive limits.
Pitfall: conflating concurrency with batching
Firing 32 concurrent requests to a stateless endpoint is not the same as a single 32-sequence batched forward pass. The gateway or model server may still schedule them separately. Check server logs or metrics for actual batched token counts.
Step 6: Implement adaptive batching
In production, requests arrive unevenly. Use a micro-batch collector: accumulate for max_wait ms or until max_size reached.
import asyncio, time
class AdaptiveBatcher:
def __init__(self, max_size=16, max_wait=0.05, send_fn=None):
self.max_size = max_size
self.max_wait = max_wait
self.send_fn = send_fn
self.pending = []
self.timer = None
async def submit(self, payload):
fut = asyncio.get_event_loop().create_future()
self.pending.append((payload, fut))
if self.timer is None:
self.timer = asyncio.create_task(self._flush_after_wait())
if len(self.pending) >= self.max_size:
await self._flush()
return await fut
async def _flush_after_wait(self):
await asyncio.sleep(self.max_wait)
await self._flush()
async def _flush(self):
if not self.pending:
return
batch, futures = self.pending, [f for _, f in self.pending]
self.pending = []
self.timer = None
try:
results = await self.send_fn([p for p, _ in batch])
for fut, res in zip(futures, results):
fut.set_result(res)
except Exception as e:
for fut in futures:
fut.set_exception(e)Tune max_wait from Step 1’s budget. If p95 budget is 300ms and S_base is 200ms, max_wait must be ≤ 100ms.
Common pitfalls
Ignoring tail latency
Averaging hides p99 spikes. A batch of 64 may be fine at p50 but OOM at p99 under long-context inputs. Always track max latency.
Over-batching on memory
KV cache size scales with batch_size × seq_len. A 7B model at 4k context may cap at batch 32; pushing 64 triggers eviction or crashes. Monitor GPU memory.
Assuming linear scaling
Throughput rarely scales linearly with batch size because decode is bandwidth-bound. If you double batch and get 1.1× throughput, stop.
Provider degradation
Providers rate-limit or degrade. If a batch call fails mid-flight, you must retry or route elsewhere. If you front your inference with a gateway such as n4n.ai, automatic fallback when a provider is rate-limited or degraded prevents the whole batch from stalling, but your code still needs to handle partial failures and idempotency.
Production checklist
- Define SLO: write p50/p95 latency and cost limits.
- Baseline: measure single-stream p50/p95 with realistic prompts.
- Sweep: run concurrency/batch sweep, log throughput and latency.
- Find knee: pick batch size where marginal throughput < 15% gain.
- Queue model: set
max_waitfrom latency headroom. - Implement: deploy adaptive batcher with size and time caps.
- Observe: track p95, KV cache occupancy, and per-token cost.
- Handle failure: add fallback for provider errors and partial batches.
The throughput latency batch size tradeoff is not a one-time setting. Traffic shape shifts, new models appear, and your knee moves. Re-sweep monthly or when p95 drifts.