The gap between tokens per second single vs concurrent requests is the single most misunderstood variable in LLM capacity planning. A provider that advertises 100 tokens/sec on a isolated benchmark may sustain half that per stream under real concurrent load, while still doubling aggregate output. You need to measure both axes before trusting any throughput ranking.
Why single-request tokens/sec lies to you
A single streaming request measures the best-case latency path: one sequence occupying a decoder with no contention for scheduling, memory bandwidth, or KV cache space. That number is real, but it describes a workload that does not exist in production unless you have exactly one user.
Autoregressive decoding is memory-bandwidth bound, not compute bound, for most practical model sizes. Each generated token requires a full weight read from HBM. With one stream, the GPU’s compute units sit idle while the memory bus delivers weights for a single batch dimension. The hardware is fundamentally underutilized.
When you launch a second request, the scheduler can often fuse the two decode steps into one larger matrix multiply. The arithmetic intensity rises, and the cost of moving weights is amortized across more sequences. This is why aggregate throughput climbs. But the individual streams now share that bus, so each sees lower tokens/sec than it did alone.
What concurrency actually changes
Batching and compute density
Transformers decode one token per step. The forward pass is a series of GEMMs against weight matrices. A single sequence gives batch size 1; the effective TFLOPS achieved is a fraction of peak. Concurrent requests let the serving stack (vLLM, TensorRT-LLM, or a provider’s custom router) pack many sequences into a single batch. Utilization improves until the batch saturates compute or memory capacity.
KV cache contention
Each active sequence caches its key/value tensors for every layer. That cache grows with sequence length and batch size. At some concurrency level, the cache pressure forces either smaller batches, eviction of older contexts, or swapping to host memory—all of which degrade per-stream speed. A 70B parameter model in fp16 already consumes ~140GB for weights alone; KV cache for long contexts is additional and can become the binding constraint.
Scheduling and provider limits
Managed endpoints enforce per-account rate limits and queue depth. A request that arrives while the batch is full waits in a pre-processing queue, inflating its wall-clock time without generating tokens. An OpenAI-compatible gateway such as n4n.ai, which fronts 240+ models with automatic fallback when a provider is degraded, can mask a single provider’s concurrency collapse if you don’t pin the route during a benchmark. Honor the client routing directive and disable fallback in your test harness to get clean numbers.
Benchmark methodology that doesn’t lie
Isolate the variable
Fix the model, the prompt length, the max tokens, and the decoding parameters (temperature, top_p). Run a single-request baseline, then run concurrency levels of 2, 4, 8, 16, 32. Use the same prompt text repeated to avoid variance from tokenization. If you care about long-context behavior, use a realistic system prompt and few-shot history.
Do not mix models mid-test. Different model architectures batch differently; comparing a 7B to a 70B under the same concurrency tells you nothing about either.
A minimal load generator
The script below fires N concurrent streaming requests against any OpenAI-compatible endpoint and reports both aggregate and per-stream tokens/sec. It uses the official openai python package; swap the base_url for your gateway.
import asyncio
import openai
import time
async def stream_one(client, prompt, max_tokens):
start = time.monotonic()
resp = await client.chat.completions.create(
model="mistral-7b-instruct",
messages=[{"role": "user", "content": prompt}],
max_tokens=max_tokens,
stream=True,
)
tokens = 0
async for chunk in resp:
if chunk.choices[0].delta.content:
tokens += 1
dur = time.monotonic() - start
return tokens, dur
async def main(concurrency, prompt, max_tokens=256):
client = openai.AsyncOpenAI(
base_url="https://api.your-gateway.com/v1",
api_key="sk-your-key",
)
tasks = [stream_one(client, prompt, max_tokens) for _ in range(concurrency)]
results = await asyncio.gather(*tasks)
total_tok = sum(r[0] for r in results)
wall = max(r[1] for r in results) # batch completes when slowest finishes
agg_tps = total_tok / wall
avg_per = sum(r[0] / r[1] for r in results) / len(results)
print(f"concurrency={concurrency} agg_tps={agg_tps:.1f} per_stream_tps={avg_per:.1f}")
asyncio.run(main(8, "Explain batch inference in three sentences."))Run this against the same model from a machine with low network jitter. Repeat three times and take the median; tail latency at high concurrency is noisy.
Measure what matters
Define two metrics explicitly:
- Aggregate tokens/sec = total tokens generated by all streams / wall-clock duration of the slowest stream.
- Per-stream tokens/sec = tokens generated by one stream / its own duration.
The tokens per second single vs concurrent requests comparison only becomes meaningful when you plot both curves against concurrency on the x-axis.
Reading the numbers: aggregate vs per-stream
The aggregate curve typically rises steeply from concurrency 1 to some knee, then flattens. The per-stream curve declines from its single-request peak, often asymptotically approaching a floor determined by memory bandwidth per sequence.
If your aggregate plateaus at concurrency 4, the serving stack is already saturated for that model on that hardware. Pushing concurrency higher only increases queue delay and worsens per-stream latency without more output. If aggregate still climbs at 32, you have headroom and can scale request parallelism to reduce cost per token.
A common mistake is reporting only the aggregate. A bulk ingestion pipeline cares about aggregate, but a chat UI where a human waits on each token cares about the per-stream number at the concurrency your frontend actually produces.
Tradeoffs: latency vs throughput
Lower per-stream tokens/sec means higher time-to-first-token and slower character rendering. Users perceive “the model got dumb” when it is actually just contended. If your product is interactive, you must cap concurrency per model instance or provision enough replicas so that steady-state concurrency stays left of the knee.
Higher aggregate throughput at higher concurrency lowers dollar cost per million tokens because you amortize fixed inference overhead. Batch jobs, eval harnesses, and offline summarization should deliberately maximize concurrency up to the plateau.
The two goals conflict. You cannot have maximal per-stream speed and maximal aggregate throughput on the same instance at the same time. The decision is a capacity allocation problem, not a model quality problem.
When to care about tokens per second single vs concurrent requests
If you are building a single-user copilot or a CLI tool, the single-request number is your spec. Benchmark it, then add a 20% safety margin for background load.
If you are building a multi-tenant API, the concurrent numbers define your autoscaling policy. Track the aggregate plateau and set your max replicas so that (requests_in_flight / replicas) <= knee_concurrency.
If you are publishing a throughput ranking, report both. A table that lists only single-request tokens/sec is advertising, not engineering. A table that lists only aggregate at concurrency 64 is equally misleading for interactive use cases.
Takeaway
Always benchmark tokens per second single vs concurrent requests as a paired measurement, on the exact model and endpoint you will ship against. Use the single-request figure to set latency expectations for interactive paths; use the aggregate curve to size fleet capacity and predict cost. Treat any published LLM speed claim that omits concurrency level as incomplete, and pin your gateway routing during tests so fallback and provider variability do not corrupt the data.