How to benchmark LLM performance: a practical methodology

Most teams guess at model speed and quality instead of measuring it. Knowing how to benchmark LLM performance properly separates reliable production systems from fragile demos. This guide gives you a reproducible methodology you can run against any OpenAI-compatible endpoint today.

Step 1: Define the metrics that matter

Before writing a single line of code, decide what “performance” means for your use case. Latency and throughput are easy to quantify; output quality is harder but cannot be ignored.

Core metrics:

• Time to first token (TTFT): milliseconds from request send to first byte of response. Users perceive this as “responsiveness” in chat UIs.
• Total latency: end-to-end request time for a complete response.
• Generation throughput: output tokens per second (tokens/s). This drives cost and perceived speed on long outputs.
• Request throughput: completed requests per second under concurrency.
• Quality score: task-specific, e.g., JSON validity, ROUGE, or LLM-as-judge.

Define a Metrics container so later steps stay structured.

from dataclasses import dataclass
import time

@dataclass
class RequestMetrics:
    model: str
    ttft_ms: float
    total_ms: float
    output_tokens: int
    input_tokens: int

    @property
    def tokens_per_sec(self) -> float:
        gen_ms = self.total_ms - self.ttft_ms
        return self.output_tokens / (gen_ms / 1000) if gen_ms > 0 else 0.0

Do not mix metrics from different payload sizes. A 10-token response and a 1,000-token response have different TTFT profiles. Segment your benchmark by output length.

Step 2: Build a representative prompt set

Synthetic “hello world” prompts will lie to you. Pull 50–200 real prompts from your production logs, or craft a JSONL file that mirrors their length and structure.

{"messages": [{"role": "user", "content": "Summarize: {long_text}"}]}
{"messages": [{"role": "user", "content": "Extract entities from: {doc}"}]}

Load them in Python:

import json

def load_prompts(path: str) -> list[dict]:
    with open(path) as f:
        return [json.loads(line)["messages"] for line in f if line.strip()]

Keep the set fixed across runs. If you change prompts, you invalidate prior comparisons. Analyze the distribution of input lengths to ensure you are not accidentally testing only trivial cases.

import statistics
def report_lengths(prompts):
    lens = [sum(len(m["content"]) for m in p) for p in prompts]
    print(f"input chars: min={min(lens)} median={statistics.median(lens)} max={max(lens)}")

Step 3: Write a minimal benchmarking harness

Use the official openai Python SDK with an async client. Point base_url at your gateway. If you point it at an OpenAI-compatible gateway such as n4n.ai, you get access to 240+ models behind one endpoint and automatic fallback when a provider is rate-limited, so you can swap model= without rewriting the harness.

from openai import AsyncOpenAI
import os

client = AsyncOpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("BASE_URL", "https://api.openai.com/v1"),
)

async def send_one(model: str, messages: list[dict]) -> RequestMetrics:
    start = time.perf_counter()
    first_token = None
    output_tokens = 0
    stream = await client.chat.completions.create(
        model=model, messages=messages, stream=True
    )
    async for chunk in stream:
        if first_token is None and chunk.choices[0].delta.content:
            first_token = time.perf_counter()
        delta = chunk.choices[0].delta.content or ""
        output_tokens += len(delta.split())  # rough token estimate
    end = time.perf_counter()
    ttft_ms = (first_token - start) * 1000 if first_token else -1
    total_ms = (end - start) * 1000
    return RequestMetrics(model, ttft_ms, total_ms, output_tokens, 0)

Always stream in production-like tests. Non-streaming hides TTFT and masks proxy buffering. For exact token counts, call without stream and read usage, or use a tokenizer locally.

Step 4: Run warm-up and isolated measurements

Cold caches and connection pools skew early numbers. Send 5 warm-up requests, discard them, then measure.

async def benchmark(model: str, prompts: list[dict], n: int = 20):
    for _ in range(5):
        await send_one(model, prompts[0])
    results = []
    for i in range(n):
        m = await send_one(model, prompts[i % len(prompts)])
        results.append(m)
    return results

Run sequentially first to get a clean latency baseline:

python -c "import asyncio, bench; print(asyncio.run(bench.benchmark('gpt-4o-mini', bench.load_prompts('prompts.jsonl'))))"

Record the machine’s region and network path. Benchmarking from a laptop on Wi-Fi introduces variance no model change can explain.

Step 5: Capture concurrency and throughput

Production traffic is parallel. Use asyncio.gather with a bounded semaphore to simulate concurrency.

import asyncio

async def benchmark_concurrent(model: str, prompts: list[dict], concurrency: int = 10, total: int = 100):
    sem = asyncio.Semaphore(concurrency)
    async def worker(idx):
        async with sem:
            return await send_one(model, prompts[idx % len(prompts)])
    start = time.perf_counter()
    results = await asyncio.gather(*[worker(i) for i in range(total)])
    elapsed = time.perf_counter() - start
    print(f"Completed {total} requests in {elapsed:.2f}s => {total/elapsed:.2f} req/s")
    return results

Watch for rate-limit errors. A gateway that honors client routing directives and forwards provider cache-control hints will let you test cache hits by sending the same prompt twice. Repeat the identical request and compare TTFT; a second call should be faster if the provider caches.

Find the saturation point: increase concurrency until latency climbs non-linearly. That knee is your real throughput limit.

Step 6: Measure output quality

Speed without correctness is useless. For structured tasks, validate JSON:

import json

def validate_json(output: str) -> bool:
    try:
        json.loads(output)
        return True
    except ValueError:
        return False

For open-ended tasks, use a stronger model as a judge. Send the original prompt, the candidate output, and a rubric to a separate evaluation call. Keep the judge prompt fixed.

async def judge(model: str, prompt: str, candidate: str) -> int:
    resp = await client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": "Score 1-5 for correctness."},
                  {"role": "user", "content": f"Prompt: {prompt}\nOutput: {candidate}"}],
    )
    return int(resp.choices[0].message.content.strip()[0])

Quality eval is itself a cost. Sample 20–30 outputs per model rather than all 200. Statistical significance matters more than coverage at this stage.

Step 7: Store, diff, and track over time

Write results to CSV with a timestamp and model version. Never rely on memory.

import csv, datetime

def save_results(model: str, results: list[RequestMetrics]):
    ts = datetime.datetime.utcnow().isoformat()
    with open("benchmarks.csv", "a", newline="") as f:
        w = csv.writer(f)
        for r in results:
            w.writerow([ts, model, r.ttft_ms, r.total_ms, r.tokens_per_sec])

If your endpoint provides per-token usage metering, log those fields to compute cost per 1k requests. An OpenAI-compatible gateway like n4n.ai includes per-token usage metering, which lets you fold cost into the same benchmark table without custom accounting.

Plot TTFT distributions with any tool you like. When you change models or providers, rerun the exact same prompt set and diff the medians. Set a threshold: if p95 latency regresses more than 15%, alert.

How to verify success

You have a working benchmark when:

1. Running the harness twice on the same model yields median TTFT within 10% variance.
2. Concurrent runs show throughput that matches your load test goals.
3. Quality scores are recorded alongside latency, not inferred.

If those hold, you know how to benchmark LLM performance in a way that survives contact with production.