n4nAI
Open console →
Benchmarks & performanceHow-toBenchmark Methodology and Measurement

How to reproduce published LLM latency benchmarks

Practical steps to reproduce LLM latency benchmarks with pinned models, fixed prompts, and a Python streaming harness for measuring TTFT and token throughput.

n4n Team3 min read765 words

Audio narration

Coming soon — every post will get a voice note here.

When you try to reproduce LLM latency benchmarks from a vendor post or paper, the numbers rarely line up. The discrepancy usually stems from unstated variables: model revision, deployment region, concurrency, and precisely what “latency” measures. The following steps show how to reproduce LLM latency benchmarks with controlled conditions and a runnable harness, so your numbers are defensible.

Step 1: Pin the exact model and provider configuration

Treat the model as a binary, not a name. “gpt-4o” or “llama-3-70b” without a date or version hash is ambiguous because providers roll updates silently. Pull the model identifier exactly as published; if the paper cites openai/gpt-4o-2024-05-13, use that string. Some providers expose a model_version field in the response body—log it.

Record the endpoint region. A request to us-east-1 and eu-west-1 can differ by 80–120 ms in connection setup alone. If the benchmark used a specific provider’s dedicated instance, note that. For multi-provider studies, an OpenAI-compatible gateway like n4n.ai addresses 240+ models through one endpoint, honors client routing directives, and forwards provider cache-control hints, letting you test cached vs uncached paths explicitly.

Create a config file:

{
  "model": "openai/gpt-4o-2024-05-13",
  "region": "us-east-1",
  "base_url": "https://api.openai.com/v1",
  "api_key_env": "OPENAI_API_KEY"
}

Store the client library version. openai-python==1.30.0 and 1.40.0 can serialize requests differently, affecting tiny but real overhead.

Step 2: Define the latency metrics you are measuring

Latency is not a single number. Publish all three:

  • TTFT (Time To First Token): from request send to first byte of response stream.
  • TPS (Tokens Per Second): generated tokens / generation time (excluding TTFT).
  • E2E (End-to-End): from send to final token.

Formulas:

TTFT = t_first_token - t_send
gen_time = t_last_token - t_first_token
TPS = output_tokens / gen_time
E2E = t_last_token - t_send

If the published benchmark only quotes “latency”, infer from context: streaming UI latency is usually TTFT; batch throughput is TPS. When you reproduce LLM latency benchmarks, report all three—otherwise the comparison is meaningless.

Step 3: Control the input and output shapes

Fix the prompt to a known token count. Use a deterministic filler that mimics real text. Do not use random words; some tokenizers compress repeated patterns differently, but stability matters more than realism for a baseline. For exactness, count tokens with the provider’s tokenizer.

import tiktoken

def make_prompt(tokens=1000):
    enc = tiktoken.get_encoding("o200k_base")
    sentence = "The quick brown fox jumps over the lazy dog. "
    ids = enc.encode(sentence)
    repeat = (tokens // len(ids)) + 1
    return enc.decode(ids * repeat)[:tokens]

PROMPT = make_prompt(1000)

Set max_tokens explicitly. Use temperature: 0 and top_p: 1. If the API supports seed, set it. Never rely on default max_tokens; some providers default to 16, others to infinity.

REQUEST_BODY = {
    "model": "openai/gpt-4o-2024-05-13",
    "messages": [{"role": "user", "content": PROMPT}],
    "max_tokens": 256,
    "temperature": 0,
    "stream": True,
}

Step 4: Write a measurement harness

Use the OpenAI Python client with streaming. Measure with perf_counter (monotonic clock). The code below captures TTFT and TPS per call and skips the first warm-up request.

import os, time, json
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def measure(prompt, max_tokens=256, trials=5):
    results = []
    for i in range(trials + 1):  # +1 warmup
        t_send = time.perf_counter()
        t_first = None
        tokens = 0
        stream = client.chat.completions.create(
            model="openai/gpt-4o-2024-05-13",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=max_tokens,
            temperature=0,
            stream=True,
        )
        for chunk in stream:
            if chunk.choices[0].delta.content:
                tokens += 1
                if t_first is None:
                    t_first = time.perf_counter()
        t_last = time.perf_counter()
        if i == 0:
            continue  # discard warmup
        results.append({
            "ttft": t_first - t_send,
            "tps": tokens / (t_last - t_first) if t_last > t_first else 0,
            "e2e": t_last - t_send,
            "tokens": tokens
        })
    return results

if __name__ == "__main__":
    prompt = make_prompt(1000)
    data = measure(prompt, max_tokens=256, trials=5)
    print(json.dumps(data, indent=2))

Run this on the same machine and network you intend to benchmark from. A laptop on Wi-Fi will not reproduce a datacenter number.

Verify success: You should see tokens equal to roughly max_tokens (minus tokenization variance). TTFT should be stable within a single-digit percentage across trials after the warm-up call. If TTFT varies by 50%, you have a network or cold-start problem, not a model latency signal.

Step 5: Account for concurrency and load

Most published LLM latency benchmarks include a concurrency parameter. If they tested 10 simultaneous requests, you must replicate that. Sequential measurements hide queueing and batching effects on the server side.

Use asyncio for concurrent streaming:

import asyncio, os, time
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])

async def single(prompt, sem):
    async with sem:
        t_send = time.perf_counter()
        t_first = None
        tokens = 0
        stream = await client.chat.completions.create(
            model="openai/gpt-4o-2024-05-13",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=256, temperature=0, stream=True,
        )
        async for chunk in stream:
            if chunk.choices[0].delta.content:
                tokens += 1
                if t_first is None:
                    t_first = time.perf_counter()
        t_last = time.perf_counter()
        return {"ttft": t_first - t_send, "e2e": t_last - t_send}

async def run_concurrent(n, prompt):
    sem = asyncio.Semaphore(n)
    tasks = [single(prompt, sem) for _ in range(n)]
    return await asyncio.gather(*tasks)

if __name__ == "__main__":
    prompt = make_prompt(1000)
    results = asyncio.run(run_concurrent(10, prompt))
    print(results)

If the published benchmark used a specific load generator (e.g., locust or k6), mirror its ramp-up pattern. Tail latency (p99) only appears under concurrency.

Step 6: Normalize across providers

When you reproduce LLM latency benchmarks that span multiple backends, provider-specific cache behavior skews TTFT. Some providers cache the prompt prefix; others charge and delay. Without explicit cache control, your reproduced numbers will mismatch a vendor who measured with warm cache.

Test both cached and uncached paths. If your gateway supports it, send cache directives:

response = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet",
    messages=[{"role": "user", "content": prompt}],
    max_tokens=256,
    extra_headers={"x-cache-control": "no-cache"},
    stream=True,
)
# inspect response.headers.get("x-cache") if the proxy exposes it

Log whether the response came from cache. This single variable explains more benchmark disputes than model speed does.

Step 7: Run, collect, and statistically analyze

Execute each step at least 20 times (excluding warm-up) to get distributions. Compute p50, p90, p99.

import statistics as st

def summarize(results):
    ttfts = [r["ttft"] for r in results]
    tpss = [r["tps"] for r in results]
    ttfts.sort()
    return {
        "ttft_p50": st.median(ttfts),
        "ttft_p90": ttfts[int(0.9*len(ttfts))],
        "ttft_p99": ttfts[int(0.99*len(ttfts))],
        "tps_median": st.median(tpss),
    }

# Example usage after collecting `results` from Step 4 or 5
print(summarize(results))

Compare to published figures. Expect variance from network path and hardware generation. If your p50 TTFT is within 20–30% of the published value and TPS tracks within 15%, you have successfully reproduce LLM latency benchmarks for practical purposes. If you are off by 2x, re-check region and cache state.

Drop outliers only with justification (e.g., a crashed connection). Record the count of dropped samples.

Step 8: Document everything for others

Publish your config: model string, region, client version, network description, prompt text, token counts, concurrency, and raw data. Without this, another engineer cannot reproduce your reproduction.

A minimal README snippet:

## Benchmark config
- Model: openai/gpt-4o-2024-05-13
- Region: us-east-1
- Client: openai-python 1.30.0
- Network: EC2 t3.medium, same AZ as endpoint
- Prompt tokens: 1000 (approx, via tiktoken)
- Max tokens: 256
- Concurrency: 1 (sequential)
- Trials: 25 + 1 warmup
- Cache: default (prompt not pre-cached)

Include the raw JSON of all trials in a companion file. When you reproduce LLM latency benchmarks with these steps, the numbers become a baseline you can defend in a design doc or a vendor negotiation. Reproducibility is a discipline, not a screenshot.

Tagsreproducibilitybenchmark-methodologylatency-benchmarkhow-to

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceHow to control for prompt length when benchmarking LLMsStep-by-step method to run a controlled prompt length llm benchmark: precise tokenization, cache disabling, length sweeps, and cross-model normalization.Benchmarks & performanceHow network hops affect LLM benchmark measurementsNetwork latency and hop count silently distort LLM benchmark numbers. This analysis shows how to measure inference performance without infrastructure bias.Benchmarks & performanceHow to benchmark LLM latency across regions and providersLearn how to benchmark LLM latency across regions with a reproducible harness measuring TTFT, token throughput, and errors from multiple clouds and providers.Benchmarks & performanceMeasuring cold start latency across LLM API providersA practical framework for measuring cold start latency llm api across providers, with honest tradeoffs in benchmark methodology and routing design.Benchmarks & performanceWhy concurrency level changes your LLM benchmark resultsConcurrency level silently skews LLM benchmark numbers. Learn why concurrency llm benchmark results diverge and how to measure throughput and latency correctly.Benchmarks & performanceWhy synthetic benchmarks miss real-world LLM latencySynthetic benchmarks vs real-world latency: why lab measurements of LLM speed fail in production, and how to measure what users actually experience.

All benchmark methodology and measurement posts →