Token throughput under concurrent load is consistently lower than the per-request numbers quoted in provider benchmarks. The gap isn’t a bug; it’s the emergent result of how inference servers schedule batched attention across limited memory bandwidth and finite KV-cache space.
The linear-scaling fallacy
Engineers often extrapolate from a single curl command: one request returned 120 tokens in 2 seconds, so 60 tok/s. Spin up 10 threads and you expect 600 tok/s aggregate. That mental model treats the GPU as a CPU running independent processes, ignoring that transformer decode is memory-bandwidth bound and globally synchronized per step.
When you actually measure token throughput under concurrent load, the curve is sublinear. Doubling concurrency rarely doubles aggregate tokens per second. At some point, adding more in-flight requests makes every request slower and the total throughput plateau or even regress.
Inside the inference server: prefill and decode
An LLM request splits into two phases:
- Prefill: process the full prompt in parallel, compute the initial hidden states and populate the KV-cache. Compute-bound, benefits from large matmuls.
- Decode: generate one token at a time, each step a small matmul constrained by memory bandwidth to fetch weights and KV-cache.
A single decode step for a batch of N sequences still requires reading the entire model weights once. Larger N amortizes weight fetch over more tokens, which is why batching raises throughput. But it also forces the scheduler to wait for the slowest sequence in the batch to finish its step, and to pad shorter sequences.
Batching wins, then loses
Continuous batching (used by vLLM, TensorRT-LLM, and others) mitigates the “wait for slowest” problem by swapping finished requests out and new ones in. Yet the scheduler still has a maximum batch size dictated by GPU memory. Once the batch is full, new requests queue.
The throughput gain from batching is real but diminishing. Going from 1 to 8 concurrent streams might lift aggregate throughput by 5x. Going from 8 to 64 might add only 1.5x more because decode steps become memory-bound on KV-cache reads rather than compute.
KV-cache contention
Each sequence occupies a slice of GPU memory for its KV-cache. At high concurrency, the cache pool fragments or exhausts. When a request can’t allocate KV-cache, the server either blocks or evicts another sequence’s cache (if prefix caching is used), forcing recomputation on the next step. That recomputation is pure waste that doesn’t produce output tokens.
Measuring it yourself
You don’t need a fancy framework to see the drop. A minimal asyncio script against an OpenAI-compatible endpoint exposes the shape:
import asyncio
import time
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="https://your-endpoint/v1", api_key="sk-...")
async def send(prompt, max_tokens=128):
t0 = time.monotonic()
resp = await client.chat.completions.create(
model="openai/gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
max_tokens=max_tokens,
)
elapsed = time.monotonic() - t0
return resp.usage.completion_tokens, elapsed
async def load_test(concurrency, total_requests):
sem = asyncio.Semaphore(concurrency)
async def bounded(i):
async with sem:
return await send(f"Write a haiku about item {i}")
t0 = time.monotonic()
results = await asyncio.gather(*[bounded(i) for i in range(total_requests)])
wall = time.monotonic() - t0
total_tokens = sum(r[0] for r in results)
return total_tokens / wall, total_tokens, wall
# Example: asyncio.run(load_test(4, 32))Run this with concurrency=1, then 4, 8, 16. Plot aggregate tok/s against concurrency. The line will bend. That bend is your real token throughput under concurrent load, not the single-stream number from docs.
Three mechanisms that cut throughput
1. Queueing delay (Little’s law)
If the server can sustain a maximum service rate μ (tokens/s) and you arrive at rate λ (tokens/s demanded), utilization ρ = λ/μ. At ρ > 0.7, average wait time grows nonlinearly. Requests that sit in the queue are not generating tokens, so wall-clock aggregate throughput includes dead time. Little’s law (L = λW) tells you that higher concurrency increases L (in-flight requests) but also W (wait), eating the perceived gain.
2. Padding and amortization loss
Batched decode pads sequences to the longest in the batch. If one request asks for 1024 tokens and nine ask for 32, the scheduler wastes compute on padded positions for the short ones every step. The effective tokens generated per weight-fetch drops. This is especially painful with heterogeneous max_tokens.
3. Cache eviction storms
With prefix caching, repeated system prompts can be reused. Under bursty concurrency, the cache fills and the eviction policy may drop a prefix that another queued request needs. That request then pays full prefill cost mid-decode, stalling the batch. The symptom: periodic throughput cliffs correlated with cache churn, not request rate.
Tradeoffs: latency vs aggregate throughput
Increasing batch size is the only lever to raise aggregate throughput on a fixed GPU. But larger batches raise per-token latency for each request because a token is emitted only after the batched step completes. For interactive chat, p99 latency matters more than total tok/s; for offline extraction, you can crank concurrency and accept slower individual jobs.
Autoscaling hides the cliff by adding replicas, but cold starts cost seconds and KV-cache warmth is lost on reschedule. A replica that just booted serves at single-stream speeds until its cache populates.
Gateway mitigation and its limits
A gateway such as n4n.ai can mitigate provider-side rate limits by automatic fallback to a healthy provider when one is degraded. That redistributes your concurrent load across backends, but the token throughput under concurrent load at any single model still follows the queueing curves above. Fallback is not a throughput multiplier; it’s a availability band-aid that prevents total stall.
Client-side routing directives (e.g., pinning to a specific provider region) can reduce tail latency by avoiding cross-region hops, but they concentrate load and may worsen batching if the target shard is small.
Takeaway
Model your LLM endpoint as a batching queue with finite KV-cache, not a parallel token factory. Load-test with the concurrency profile of your real traffic, measure aggregate tok/s at each level, and set SLOs from the bent curve—not from single-request benchmarks. When throughput drops, add replicas or shed load; tune batch sizes only if you can tolerate the latency cost. Gateways and fallback buy you headroom against provider outages, but they do not suspend the laws of queueing.