How concurrency affects latency: 10 vs 100 requests

The concurrency effect on LLM API latency is non-linear and frequently misunderstood. Sending 10 requests in parallel behaves very differently from sending 100, not just in aggregate throughput but in the latency distribution each individual call experiences. This analysis uses queueing theory, provider limits, and gateway behavior to show where the cliff is and how to engineer around it.

Why LLM inference breaks naive scaling

LLM servers do not process requests as isolated tasks. They batch token decoding across sequences to keep GPU matmul units saturated. A single replica can hold a batch of dozens of active sequences, but KV cache memory caps that batch size hard.

When you send 10 concurrent requests, the scheduler fits them into one or two batches. Each request waits for its position in the batch plus network round-trip. At 100 concurrent, the scheduler either queues requests or rejects them with 429. The concurrency effect on LLM API latency at that point is dominated by queue wait, not by compute time.

Batching under the hood

Autoregressive decoding generates one token per step per sequence. The cost per step is roughly constant for a batch up to the memory limit. Doubling batch size doubles tokens per second but adds marginal latency per request only when the batch is full. Past the limit, requests wait for the next free slot.

A queueing model that predicts the cliff

Treat a provider model endpoint as a pool of c identical workers, each with service rate μ (requests per second). Your arrival rate λ is concurrency divided by average request duration. In an M/M/c queue, average wait grows as λ approaches c*μ.

The resident wait formula is complex, but the intuition is simple: utilization ρ = λ / (c*μ). When ρ < 0.7, queue stays short. When ρ > 0.9, wait time explodes.

import math

def mmc_utilization(lam, mu, c):
    return lam / (c * mu)

def mm1_wait(lam, mu):
    if lam >= mu:
        return float('inf')
    return 1/(mu - lam) - 1/mu

# single worker, 2s service time -> mu=0.5
print(mm1_wait(0.45, 0.5))  # ~18s wait at 90% util

Real endpoints are multi-worker, but the same hyperbolic curve applies per replica. The concurrency effect on LLM API latency tracks that curve.

What 10 concurrent actually looks like

Assume a provider replica services 5 requests/sec aggregate when batched, and you send 10 short requests (100 tokens each). The replica batches them; p50 latency stays near 1–3 seconds, p95 within 5 seconds. Throughput gains are near-linear: 10 parallel finishes in roughly the time of 2–3 serial calls.

The concurrency effect on LLM API latency here is mild because λ is far below c*μ. You are in the safe zone.

What 100 concurrent does

Push to 100 requests against the same pool. If the provider caps concurrent batches at 32, 68 requests sit in queue. Clients see p95 climb to tens of seconds; many receive 429 Too Many Requests.

Retries amplify load. Naive exponential backoff with jitter still adds requests, worsening the concurrency effect on LLM API latency for every tenant sharing the model. Public provider documentation commonly shows RPM tiers from low hundreds to low thousands; 100 concurrent with sub-second think time easily exceeds those.

Measure it yourself

Don’t trust guesses. Run a concurrency sweep against any OpenAI-compatible endpoint. The script below fires N simultaneous chat completions and records wall-clock per call.

import asyncio, aiohttp, time

API_URL = "https://api.openai.com/v1/chat/completions"
API_KEY = "sk-..."  # load from env
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

async def one_request(session, payload):
    start = time.monotonic()
    async with session.post(API_URL, json=payload, headers=HEADERS) as resp:
        await resp.json()
        return time.monotonic() - start

async def sweep(concurrency, total):
    payload = {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "say hi"}]}
    async with aiohttp.ClientSession() as session:
        tasks = []
        for _ in range(total):
            tasks.append(one_request(session, payload))
            if len(tasks) >= concurrency:
                await asyncio.gather(*tasks)
                tasks = []
        if tasks:
            await asyncio.gather(*tasks)

# asyncio.run(sweep(10, 100))
# asyncio.run(sweep(100, 100))

Run both sweeps and compare p95. An inference gateway like n4n.ai masks some of the pain by automatically failing over when a provider is rate-limited or degraded, but the underlying queue wait still surfaces as latency spikes on the client side.

Provider rate limits are the real constraint

Providers publish RPM and TPM. At 100 concurrent with 200ms client think time, you generate 30,000 RPM—easy to exceed a 10,000 RPM limit. The gateway returns 429; your code must shed load or degrade.

Honoring cache-control hints helps: repeated system prompts with cache_control: ephemeral let the provider reuse KV cache, reducing effective service time μ. That shifts the cliff outward. Per-token metering lets you attribute spend precisely when you tune batch sizes.

Throughput vs tail latency tradeoff

Batching 100 requests improves token throughput per GPU hour—providers want this. But your individual user waits longer. For interactive chat, keep concurrency per user low (1–4) and use async to overlap network IO, not compute.

For offline extraction over 10k docs, crank concurrency to 50–100 and accept p95 of 30s. The concurrency effect on LLM API latency is acceptable when no human blocks on the response.

Designing the client

Cap concurrent requests with a semaphore:

sem = asyncio.Semaphore(20)

async def bounded_request(session, payload):
    async with sem:
        return await one_request(session, payload)

Pair with jittered backoff that respects Retry-After. Gateways that forward provider cache-control and meter tokens per call let you attribute spend and reuse KV state across retries.

Decisive takeaway

The concurrency effect on LLM API latency is manageable at 10, dangerous at 100 unless you control the serving layer. Compute your provider’s c*μ from rate limits and batch size, keep λ under 70% of that, and use a gateway with fallback to absorb spikes. For user-facing features, limit parallelism and optimize batch size; for batch jobs, push concurrency but monitor p95 and 429 rates. Latency cliffs are predictable—engineer before you hit them.