How to measure p50, p95, and p99 latency for LLM APIs

Knowing how to measure p50 p95 p99 latency for LLM APIs separates a real capacity plan from a guess. Most teams log only average response time and then get paged at 2 a.m. when the tail explodes. This guide gives you a reproducible method and code to capture percentiles that match production behavior, not a vendor’s marketing slide.

Step 1: Define the latency metrics that matter

LLM inference has two distinct latency surfaces. Time-to-first-token (TTFT) measures how long until the client receives the first streamed chunk. Total request latency measures the full duration until the final token arrives. If you only record the end-to-end call time, you hide the interactive lag that users feel while typing.

Decide which metric your SLO targets. For chat UIs, TTFT under 500 ms is often the p95 goal; for batch extraction, total latency dominates. Always record both.

A third metric, tokens per second, is derived, not measured directly. Compute it only after you have solid latency numbers, and never use it as a substitute for the raw percentiles.

Step 2: Build a minimal streaming probe

Use the OpenAI Python client with stream=True. The first chunk timestamp gives TTFT; the closing timestamp gives total latency. Wrap the call in perf_counter to avoid time.time drift from NTP adjustments.

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

def probe(prompt: str) -> dict:
    start = time.perf_counter()
    ttft = None
    chunks = 0
    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    for chunk in stream:
        chunks += 1
        if ttft is None and chunk.choices[0].delta.content:
            ttft = time.perf_counter() - start
    total = time.perf_counter() - start
    return {"ttft": ttft, "total": total, "chunks": chunks}

Run this once to confirm your credentials and network path. If ttft is None, the model returned an empty delta; handle that by checking chunk.choices[0].finish_reason before breaking. The point is to capture the moment a human would see pixels on screen, not when the API handshake finished.

Step 3: Scale to realistic concurrency

A single sequential loop understates tail latency because real traffic arrives in parallel. This is the core of how to measure p50 p95 p99 latency for LLM APIs under load rather than in a vacuum. Use a thread pool to issue many requests at once.

import concurrent.futures
import random

PROMPTS = ["Summarize: " + "x" * 200 for _ in range(20)]

def run_batch(n_workers: int, n_requests: int):
    samples = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=n_workers) as ex:
        futures = [ex.submit(probe, random.choice(PROMPTS)) for _ in range(n_requests)]
        for f in concurrent.futures.as_completed(futures):
            samples.append(f.result())
    return samples

Set n_workers to match your production concurrency profile. If you serve 10 RPS with an average 2-second generation, you have ~20 inflight requests; use n_workers=20. Do not crank to 200 unless you intend to measure saturation collapse. The p99 you get at saturation is a different number than the p99 at steady state, and conflating them is a classic reporting error.

Step 4: Collect and store raw samples

Never compute percentiles on the fly and discard the raw numbers. Write each sample to JSONL so you can re-aggregate later with different methods.

import json

def save(samples, path="latency_raw.jsonl"):
    with open(path, "w") as f:
        for s in samples:
            f.write(json.dumps(s) + "\n")

A sample record looks like:

{"ttft": 0.212, "total": 1.834, "chunks": 142}

Keep the chunk count; it lets you spot runs where the model truncated early and skewed total latency. If you later add a new model to the comparison, the raw file is your only defense against “we lost the original data.”

Step 5: Compute percentiles without lying to yourself

The naive np.percentile(data, 95) uses linear interpolation between closest ranks. For SLO reporting, the nearest-rank method is stricter and matches “95% of requests were at or below X”. Use this:

import numpy as np

def nearest_rank_percentile(data: list[float], p: float) -> float:
    if not data:
        raise ValueError("empty")
    sorted_d = np.sort(data)
    rank = int(np.ceil(p / 100.0 * len(sorted_d)))
    rank = min(rank, len(sorted_d))
    return sorted_d[rank - 1]

samples = [s["ttft"] for s in samples]
p50 = nearest_rank_percentile(samples, 50)
p95 = nearest_rank_percentile(samples, 95)
p99 = nearest_rank_percentile(samples, 99)
print(f"TTFT p50={p50*1000:.1f}ms p95={p95*1000:.1f}ms p99={p99*1000:.1f}ms")

If you need to compare against a dashboard that uses interpolated percentiles, compute both and note the difference. For p99 with fewer than 1000 samples, the nearest-rank value jumps in coarse steps; collect more data before trusting it. A rule of thumb: you need at least 100 samples at the target percentile to see it move smoothly, so 1000 total samples is the floor for p99 work.

Step 6: Control confounding variables

Latency measurements are only valid if the test conditions match production. Warm up the model endpoint for at least 20 calls before recording; cold container starts can add seconds that are not representative. Run from the same cloud region as your users, not your laptop on hotel Wi-Fi.

If you front requests with a gateway, provider fallback will mask the tail. For example, when measuring against n4n.ai, send a routing directive that pins a single upstream provider so automatic fallback does not silently reroute a slow request to a faster one mid-benchmark. The same applies to any load balancer with health-based evasion.

Also pin the model version. Model aliases like gpt-4o can shift underneath you; use a dated snapshot if the API exposes one. Keep the prompt length fixed across runs—varying it changes token count and therefore total latency, polluting your trend line.

Step 7: Verify your measurement

A benchmark you cannot reproduce is an anecdote. Run the batch twice, five minutes apart. The p95 and p99 values should land within 10–15% of each other. If they swing wildly, you either have too few samples or noisy neighbors on shared inference hardware.

Test your harness against a mock server to confirm the math. A local Flask app that sleeps 100 ms then streams ten chunks with 10 ms gaps should yield TTFT near 100 ms and total near 200 ms.

from flask import Flask, Response
import time

app = Flask(__name__)

@app.route("/v1/chat/completions", methods=["POST"])
def mock():
    def gen():
        time.sleep(0.1)
        for _ in range(10):
            yield '{"choices":[{"delta":{"content":"x"}}]}\n'
            time.sleep(0.01)
    return Response(gen(), mimetype="application/json")

Point your base_url at this mock and run probe. If it reports 0 ms, your timestamp capture is broken. Check monotonicity: p50 <= p95 <= p99 must hold. If not, you sorted the wrong field or mixed TTFT and total.

Step 8: Automate and track over time

Percentiles drift when providers update kernels or your prompt length changes. Schedule the batch nightly with a fixed prompt set and push the results to a time-series store.

#!/usr/bin/env bash
set -euo pipefail
python bench.py --workers 20 --requests 500 --out latency_$(date +%F).jsonl
python report.py latency_$(date +%F).jsonl >> latency_trend.csv

Alert when p99 TTFT rises more than 30% week-over-week. That catches provider regressions before your users do. Store the raw JSONL alongside the summary so you can re-run with nearest-rank vs interpolated if a debate erupts in the postmortem.

What good looks like

After following these steps, you will have a JSONL file of raw latencies, a printed table of p50/p95/p99 for both TTFT and total, and a repeatable command. Share the exact prompt, worker count, and region with any number you publish. That is how to measure p50 p95 p99 latency for LLM APIs in a way that survives a postmortem.

If you skip the concurrency step and report single-threaded numbers, you will underestimate tail latency by 2–5x under load. If you skip nearest-rank, you will understate the worst-case experience. Do both, and the p99 you cite will be the p99 your users actually eat.