Measuring p99 latency under load for LLM inference

Getting accurate p99 latency measurement under load for LLM inference is harder than a single curl timing. Tail latency hides behind streaming token generation, provider rate limits, and variable prompt preprocessing, so your test must reproduce concurrency and output shapes or the number is fiction. This guide walks through building a load generator, capturing the right timestamps, and computing a defensible p99.

Step 1: Define a realistic workload profile

Don’t load test with a 5-token prompt and max_tokens=1. Production traffic arrives as a distribution, not a constant. Capture the shape first.

Define three parameters: concurrent in-flight requests, input token size, and output token size. A simple JSON spec keeps the test reproducible:

{
  "concurrency": 50,
  "prompt_tokens": 120,
  "max_output_tokens": 256,
  "duration_seconds": 300,
  "model": "gpt-4o-mini"
}

If you lack real traces, use a log-normal distribution for prompt length. Most chat prompts cluster around 50–200 tokens with a long tail. The point is to avoid a synthetic workload that flatlines the prefill cache and makes every request look identical. Identical requests let the provider cache everything and hide the p99 you actually care about.

Step 2: Build a latency-aware client

You need raw per-request timestamps, not just aggregate throughput. A single aggregated average hides the worst 1%. Below is a minimal asyncio client that hits an OpenAI-compatible endpoint, measures end-to-end wall time, and records samples. It uses a semaphore to bound concurrency and drains the response body fully.

import asyncio, aiohttp, time, json

async def send_one(session, url, payload, results):
    start = time.perf_counter()
    async with session.post(url, json=payload) as resp:
        await resp.read()  # full body, streaming or not
    results.append(time.perf_counter() - start)

async def run(concurrency, url, payload, duration):
    results = []
    async with aiohttp.ClientSession() as session:
        deadline = time.perf_counter() + duration
        sem = asyncio.Semaphore(concurrency)
        async def worker():
            while time.perf_counter() < deadline:
                async with sem:
                    await send_one(session, url, payload, results)
        await asyncio.gather(*[worker() for _ in range(concurrency)])
    return results

Connection pooling matters: aiohttp.ClientSession reuses TCP/TLS connections by default. If you instantiate a new session per request, you measure handshake cost, not inference. Set limit on the connector if you push past 100 concurrency per process.

If you route through a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint across 240+ models with automatic fallback, you can keep this client unchanged while shifting the model field to compare backends without rewriting instrumentation.

Step 3: Separate time-to-first-token from total latency

For streaming endpoints, p99 latency measurement under load must report two numbers: TTFT (time to first byte of the stream) and total completion latency. Averaging them hides the tail.

Modify the client to capture TTFT explicitly:

async def send_stream(session, url, payload, results):
    start = time.perf_counter()
    ttft = None
    async with session.post(url, json=payload) as resp:
        async for chunk in resp.content.iter_chunked(1024):
            if ttft is None:
                ttft = time.perf_counter() - start
    total = time.perf_counter() - start
    results.append((ttft, total))

Run both measurements. The p99 of TTFT tells you about prefill and queueing under contention; p99 of total tells you about decode throughput when the GPU is saturated. A system can have great TTFT and terrible total latency because it accepts too many concurrent decodes and thrashes the KV cache.

Step 4: Execute the load test from multiple hosts

A single client machine caps at its own CPU and NIC limits. If you need 500+ concurrency, launch workers in containers and merge results.

docker run --rm -v $PWD:/app -w /app python:3.12 \
  python load_gen.py --concurrency 100 --duration 300

Run three independent workers and concatenate the output JSON. Do not trust a p99 computed from a single process that itself is CPU-saturated; you’ll measure the client’s event loop, not the server. Watch the worker’s CPU with top—if it’s above 80%, split the load.

Network proximity also matters. Testing from the same region as the inference box but shipping from another in production means cross-region TLS handshake alone can add 100ms to p99. Run at least one worker in the same cloud region you deploy in.

Step 5: Compute p99 from raw samples correctly

Sort the list, pick the 99th percentile index. No pandas needed:

def p99(samples):
    s = sorted(samples)
    if not s:
        return 0.0
    idx = max(0, int(0.99 * len(s)) - 1)
    return s[idx]

latencies = [0.2, 0.3, 0.5, 1.2, 4.1]
print(p99(latencies))

If you batch requests in windows (e.g., per-minute), compute p99 per window, then look at the worst window. Aggregating all samples across a 5-minute test masks a 30-second spike that paged your on-call. The nearest-rank method above is conservative; linear interpolation is fine too, but pick one and document it.

A p99 derived from fewer than 1,000 samples is statistical noise. At 50 concurrency over 300 seconds with ~200ms latency, you’ll collect ~75,000 requests—plenty.

Step 6: Validate the number and set an SLO

Success means the p99 latency measurement under load is stable across repeated runs. Run the test, wait 10 minutes, run again. If run A gives 2.1s and run B gives 8.4s, your system is not load-tested; it’s lottery-tested.

Set an SLO like: p99 total latency < 3s at 50 concurrent for model X. Alert when two consecutive windows breach. TTFT SLO might be < 800ms at same load. These numbers come from your product’s tolerance, not from a provider’s marketing.

Step 7: Automate and correlate with cost

Put the script in CI as a nightly job against a staging endpoint. Use per-token usage metering (many gateways return this in headers) to tag each sample with cost. A p99 that jumps while token cost stays flat points to a regression in routing or cache hit rate, not provider pricing.

n4n.ai honors client routing directives and forwards provider cache-control hints, so you can toggle cache_read in the request to measure the delta of prompt caching on tail latency directly. That isolates caching impact from raw model speed.

Common mistakes

• Measuring only mean latency. The mean of a 100ms and 10s request is 5s, which describes neither.
• Ignoring stream consumption. Closing the connection after the first token and calling it “latency” undercounts decode time by orders of magnitude.
• Using time.time() instead of time.perf_counter(). The former jumps with NTP corrections and skews sub-second measurements.
• Testing from the same region as the inference box but shipping from another. Cross-region TLS alone can add 100ms to p99.
• Treating provider rate-limit errors as latency. If 5% of requests return 429, your p99 is undefined until you handle backoff and recount.

Verify success

You have a valid p99 latency measurement under load when:

1. Raw samples count > 10,000 (enough to make the 99th percentile meaningful).
2. Three independent runs produce p99 within 10% of each other.
3. TTFT p99 and total p99 are reported separately.
4. The load generator’s own CPU usage stayed below 70% on each worker.
5. Errors (5xx, 429) are logged separately and excluded from the latency set, but their rate is reported.

Anything less is a guess with a chart attached.