Why concurrency level changes your LLM benchmark results

Most published concurrency llm benchmark results are not comparable because the concurrency level is either omitted or set to a value that matches no production workload. A single serialized request hides queueing and batching effects; a thousand parallel streams hides the saturation point where latency explodes. If you care about real throughput, you have to treat concurrency as the independent variable, not a footnote.

What concurrency actually measures

Concurrency is the count of in-flight requests at any moment. It is not requests per second. The two relate through latency: average concurrency ≈ throughput (tokens/sec or req/sec) × average latency. Fix one, and the others are coupled.

When you run a benchmark at concurrency 1, you measure a single request’s round trip with no contention for GPU schedulers, no batch formation, and no queue ahead of you. At concurrency 32, the serving stack packs your requests with others into the same forward pass, amortizing overhead. That changes both latency distribution and token throughput per watt.

Ignore concurrency and you cannot interpret any other number. A latency of 800 ms is excellent at concurrency 64 and terrible at concurrency 1.

The two failure modes in common benchmarks

Serialized benchmarks hide queueing

The default example in docs is a for-loop sending one completion, awaiting it, repeating. This yields a clean p50 but tells you nothing about how the system behaves when many users hit it simultaneously.

Worse, some providers use speculative decoding or continuous batching that only engages under load. Your serialized test reports the unbatched code path. If you ship a chatbot expecting 20 simultaneous sessions, the serialized number is a lie about cost and speed.

Unbounded concurrency hides saturation

The opposite mistake is spawning asyncio tasks without limit, then reporting total tokens generated. This maximizes throughput on the chart but masks the point where the system falls over. You will see p99 latency climb to minutes, 429 errors, or silent request drops.

A number like “1.2M tokens/sec aggregate” is meaningless if the tail latency makes the model unusable for interactive use. Unbounded tests also trigger provider rate limits, so the results reflect throttling policy, not model speed.

Queueing dynamics for LLM inference

LLM serving is not a simple M/M/1 queue, but the intuition holds: as arrival rate approaches service capacity, wait time grows nonlinearly. With continuous batching, the scheduler collects requests each decoding step. Low concurrency leaves GPU tensor cores idle between steps. Raising concurrency fills those gaps, pushing utilization up.

The limit is memory, not compute. Each request allocates a KV cache proportional to context length. When the aggregate KV cache exceeds VRAM, the server either evicts caches (raising recompute) or preempts requests. That is the knee in the curve: latency jumps, throughput plateaus or drops.

Thus the useful benchmark answers: where is the knee for my prompt shape?

Designing a benchmark that controls concurrency

You need a harness that holds N workers active for a fixed duration, not just fires N and stops. Below is a minimal asyncio pattern against any OpenAI-compatible API.

import asyncio, time, openai

client = openai.AsyncOpenAI(
    base_url="https://api.openai.com/v1",  # swap for your endpoint
    api_key="YOUR_KEY",
)

async def worker(queue: asyncio.Queue, results: list):
    while True:
        item = await queue.get()
        if item is None:
            queue.task_done()
            break
        start = time.monotonic()
        try:
            await client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": "Reply with a short sentence."}],
                max_tokens=32,
            )
            results.append(time.monotonic() - start)
        except Exception:
            results.append(-1)
        finally:
            queue.task_done()

async def load_test(concurrency: int, total_requests: int):
    queue = asyncio.Queue(maxsize=concurrency)
    results = []
    workers = [asyncio.create_task(worker(queue, results)) for _ in range(concurrency)]
    for i in range(total_requests):
        await queue.put(i)
    await queue.join()
    for _ in workers:
        await queue.put(None)
    await asyncio.gather(*workers)
    return results

# Run: latencies = asyncio.run(load_test(concurrency=16, total_requests=200))

This keeps exactly concurrency requests in flight until the workload exhausts. Vary the first argument across powers of two and record p50, p95, and total tokens sampled (from the response usage field).

A config file makes runs repeatable:

{
  "model": "anthropic/claude-3.5-sonnet",
  "concurrency_levels": [1, 4, 8, 16, 32, 64],
  "prompt_tokens": 150,
  "max_tokens": 128,
  "total_requests_per_level": 300
}

Reading the curves: latency vs throughput

Plot p50 and p95 latency on the Y axis against concurrency on the X axis. Throughput (tokens/sec) should rise steeply, then flatten. The honest report shows both.

Early on, latency stays near constant because the GPU is underutilized. At the knee, p95 begins rising faster than p50—a sign of queueing. Past the knee, p50 itself climbs as every request waits for KV memory.

Do not publish only the best throughput point. A concurrency llm benchmark result that says “32 is optimal” is only true for that prompt size and that hardware. Your production mix differs.

Tradeoffs: batching, KV cache, and provider limits

Higher concurrency improves batching efficiency but increases contention for prefix cache. If you send many requests with the same system prompt, a gateway or server can reuse KV cache blocks—but only if those blocks survive. At high concurrency, eviction may discard them, negating the win.

Provider rate limits add another axis. Running this against a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, lets you attribute latency to the model rather than to provider throttling. You still hit model-level saturation, but you remove the variable of a single vendor’s 429 policy.

Per-token metering (which such gateways forward) also lets you compute cost per effective throughput, not just raw speed.

Honest tradeoffs of methodology

Fixing concurrency low gives you clean latency but underestimates real-world throughput by 5–10x. Fixing it high shows max aggregate speed but hides the latency tax your users pay. There is no universal “right” concurrency; there is only the concurrency your service actually runs at.

If you measure only one level, pick the one matching your p95 simultaneous sessions. If you cannot know, publish the curve from 1 to your infrastructure limit. Omitting the level makes the number useless to everyone else.

Decisive takeaway

Always report concurrency alongside latency percentiles and throughput. Build a harness that holds concurrency constant, sweep it geometrically, and find the knee. Any concurrency llm benchmark results that do not state the level, the prompt shape, and the tail latency are not results—they are anecdotes. Measure the system you ship, not the system that looks good in a chart.