A guide to building your own LLM latency test suite

Most teams measure LLM speed with a stopwatch around a single curl call and call it done. If you plan to build llm latency test suite that survives contact with production traffic, you need to capture streaming behavior, concurrency, and provider variability. This guide lays out an ordered path from defining metrics to analyzing percentiles, with code you can lift into your own repo.

Define the latency metrics that matter

Latency is not one number. For interactive chat, time to first token (TTFT) drives perceived responsiveness. For bulk extraction, total request time and tokens per second (TPS) matter more. Pick the metric that matches your user’s pain.

TTFT measures the gap between sending the request and receiving the first content chunk. TPS is the generation rate after that first token. Total latency is the sum, but streaming hides the split. Always record both TTFT and completion time separately.

from openai import OpenAI
import time

client = OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")

start = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain TCP slow start"}],
    stream=True,
)
first_token_ts = None
token_count = 0
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if not delta:
        continue
    if first_token_ts is None:
        first_token_ts = time.perf_counter()
    token_count += 1
end = time.perf_counter()

ttft_ms = (first_token_ts - start) * 1000
total_ms = (end - start) * 1000
tps = token_count / (end - first_token_ts) if first_token_ts else 0
print(f"TTFT: {ttft_ms:.1f}ms, Total: {total_ms:.1f}ms, TPS: {tps:.1f}")

Use time.perf_counter()—it is monotonic and unaffected by system clock adjustments. Token counting from deltas is approximate; if the API returns usage on the final chunk, prefer that for TPS.

Pick representative models and routing

Your test must mirror your real routing topology. If you call multiple providers, the build llm latency test suite needs to exercise primary, fallback, and cache paths. A gateway that honors client routing directives (e.g., n4n.ai) will forward cache-control hints and fall back on degradation; your test should simulate those paths explicitly rather than assuming a single upstream.

Don’t test only the happy path. Force a fallback by sending a routing directive that points to a degraded or secondary provider, then measure the penalty.

{
  "model": "auto-router",
  "route": {"prefer": ["provider-a"], "fallback": ["provider-b"]},
  "cache_control": {"type": "ephemeral"}
}

If you run a single model, still vary the system prompt prefix to test cached vs uncached prefixes if the provider supports prompt caching.

Instrument concurrency and load

Single-request numbers hide queueing and contention. Production traffic is concurrent; your suite must be too. Use an async client to fire N in-flight streams and capture per-request latency.

import asyncio
from openai import AsyncOpenAI

async def timed_request(client, prompt):
    start = time.perf_counter()
    stream = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    async for chunk in stream:
        _ = chunk.choices[0].delta.content
    return time.perf_counter() - start

async def main(concurrency, prompts):
    client = AsyncOpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
    tasks = [timed_request(client, p) for p in prompts[:concurrency]]
    return await asyncio.gather(*tasks)

results = asyncio.run(main(20, ["Explain quicksort"] * 100))

Scale concurrency to your p95 production fan-out, not your laptop’s core count. For sustained load, wrap this in a loop with a fixed request rate using asyncio.sleep or a proper load tool like Locust with a custom client.

Control for cache and cold starts

Provider prompt caching can cut TTFT substantially on repeated prefixes. Your build llm latency test suite must separate cached from uncached runs. Send a warm-up request with the same prefix, then measure both states.

Cold starts on serverless inference add latency spikes. If you test sporadically, you overestimate steady-state. Run a warm-up phase of at least 50 requests before collecting samples. Keep the test runner in the same region as your production inference client—cross-continent RTT alone can be 100ms+.

Record metadata with every sample: model, region, cached flag, concurrency level, prompt length. Without this, you cannot explain outliers.

Collect percentiles, not averages

Averages hide tail pain. Compute p50, p90, p95, p99 from raw samples.

import numpy as np

latencies_ms = np.array(results) * 1000
for p in [50, 90, 95, 99]:
    print(f"p{p}: {np.percentile(latencies_ms, p):.1f}ms")

Store raw samples in a structured log or timeseries DB. Plot the distribution; a bimodal spread usually means cache hits vs misses or fallback triggering. Set alerts on p95 regression rather than mean.

Common pitfalls

• Treating TTFT as total latency. Users see the first token fast but may wait long for completion on long outputs.
• Single short prompt. Real queries vary; generation time dominates at 1k+ output tokens. Use a mix of lengths.
• Ignoring network egress. A CI runner in another continent biases every number. Co-locate the harness.
• Not accounting for rate limits. When you hit 429, your “latency” becomes retry backoff. Simulate the limit or request headroom.
• Forgetting per-token metering. If you route through a gateway with per-token usage metering, the accounting overhead is negligible but the billing dimensions matter for cost-latency tradeoffs.
• No warm-up. Cold prefixes and cold containers produce scary numbers that never appear in steady state.

Tradeoffs in coverage

Exhaustive testing across hundreds of models is wasteful. Prioritize the handful of models that serve most of your traffic. Add periodic spot-checks for the long tail.

If you build llm latency test suite with every dimension (model × region × cache × concurrency), the matrix explodes. Use orthogonal sampling or random selection to keep runtime under an hour. Accept that you are measuring a moving target—providers shift infrastructure monthly.

Make it continuous

Wire the suite into CI as a scheduled job, not a one-off script. Alert when p95 regresses more than 10% week-over-week. Keep the harness code next to your inference client so it evolves with your routing logic.

Start narrow: one model, one region, streaming TTFT and TPS. Expand only when data shows variance worth tracking. That discipline is the difference between a latency dashboard and a science project.