How to benchmark LLM latency across regions and providers

If you ship LLM features to users in multiple geographies, you need to benchmark llm latency across regions before you trust a provider’s marketing. A single curl from your laptop tells you nothing about what a user in Singapore or Frankfurt experiences. This guide walks through a repeatable harness that measures time-to-first-token, inter-token latency, and total request time from multiple cloud regions against multiple providers.

Step 1: Define the metrics that matter

Latency is not one number. Break it into three signals:

• Time to first token (TTFT): milliseconds from request send to first streamed chunk. This drives perceived responsiveness.
• Tokens per second (TPS): generated tokens divided by generation time (last token minus first token).
• Total request latency: TTFT plus full generation time, plus any network close overhead.

Also track error rate and HTTP status distribution. A provider with great median TTFT but 5% timeouts is worse than one with stable, slightly higher numbers.

When you benchmark llm latency across regions, record all four per region per model. Never average raw latency across regions; physical distance dominates network jitter.

Step 2: Select regions and provider endpoints

Pick at least three cloud regions that match your user base: for example us-east-1, eu-central-1, and ap-southeast-1. Deploy identical probe containers to each.

For providers, you have two options:

1. Call each provider’s regional endpoint directly (e.g., api.openai.com, eu.api.anthropic.com).
2. Use a gateway that normalizes the interface.

A gateway such as n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and applies automatic fallback when a backend is degraded. You can still pin a specific provider per request via routing directives to isolate its latency. That collapses your client code to one base URL while preserving per-provider measurement.

Define a config file listing targets:

{
  "regions": ["us-east-1", "eu-central-1", "ap-southeast-1"],
  "targets": [
    {"name": "direct-openai", "base_url": "https://api.openai.com/v1", "model": "gpt-4o-mini"},
    {"name": "gateway-pinned", "base_url": "https://api.n4n.ai/v1", "model": "anthropic/claude-3.5-sonnet", "route": {"provider": "anthropic"}}
  ],
  "prompt": "Explain TCP congestion control in one paragraph."
}

Step 3: Build a minimal async probe

Use Python with the official openai SDK. Stream completions and timestamp chunks.

import asyncio, time, json
from openai import AsyncOpenAI

async def probe(base_url, api_key, model, prompt, route=None):
    client = AsyncOpenAI(base_url=base_url, api_key=api_key)
    extra = {"extra_headers": {"x-n4n-route": json.dumps(route)}} if route else {}
    start = time.perf_counter()
    first_token = None
    tokens = 0
    try:
        stream = await client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            stream=True,
            **extra
        )
        async for chunk in stream:
            if chunk.choices[0].delta.content:
                if first_token is None:
                    first_token = time.perf_counter()
                tokens += 1
        end = time.perf_counter()
        ttft = (first_token - start) * 1000
        gen = (end - first_token) * 1000
        tps = tokens / (gen / 1000) if gen > 0 else 0
        return {"ttft_ms": ttft, "tps": tps, "total_ms": (end-start)*1000, "tokens": tokens, "error": None}
    except Exception as e:
        return {"ttft_ms": None, "tps": 0, "total_ms": None, "tokens": 0, "error": str(e)}

This returns structured timing. Run it inside an async loop with concurrency limits to avoid self-throttling.

Step 4: Run probes from each region

Package the script into a container. Deploy the same image to each selected region using your CI:

for region in us-east-1 eu-central-1 ap-southeast-1; do
  aws ecr get-login-pipeline --region $region
  docker tag probe:latest $ACCOUNT.dkr.ecr.$region.amazonaws.com/probe:latest
  docker push $ACCOUNT.dkr.ecr.$region.amazonaws.com/probe:latest
  # launch task, e.g., ECS Fargate or Lambda
done

Inside the container, iterate over targets and run N warm-up calls then M measured calls (e.g., 5 warm-up, 20 measured). Warm-ups defeat cold DNS and connection pooling artifacts.

async def run_matrix(config, region):
    results = []
    for target in config["targets"]:
        for i in range(25):
            r = await probe(target["base_url"], KEY, target["model"], config["prompt"], target.get("route"))
            r["region"] = region
            r["target"] = target["name"]
            r["iter"] = i
            results.append(r)
            await asyncio.sleep(0.2)
    return results

Write results as JSON lines to stdout or to an object store prefix keyed by region.

Step 5: Store and aggregate results

Do not rely on local CSVs. Emit JSONL and aggregate with a separate process:

{"region":"eu-central-1","target":"gateway-pinned","ttft_ms":412.3,"tps":68.1,"total_ms":1980.2,"tokens":142,"error":null}

Compute percentiles per region/target:

import statistics, json
rows = [json.loads(l) for l in open("results.jsonl")]
def pct(vals, p):
    return statistics.quantiles(vals, n=100)[p-1]
by_key = {}
for r in rows:
    if r["error"]: continue
    key = (r["region"], r["target"])
    by_key.setdefault(key, []).append(r)
for key, items in by_key.items():
    ttfts = [i["ttft_ms"] for i in items]
    print(key, "p50", pct(ttfts,50), "p95", pct(ttfts,95))

When you benchmark llm latency across regions, plot p95 TTFT on a bar chart per region. The shape should follow round-trip distance from the region to the provider’s datacenter.

Step 6: Validate and alert

A benchmark is only useful if it is trustworthy. Verify success with these checks:

• Repeatability: Run the matrix three times. Median TTFT per region should vary less than 10% between runs.
• Error budget: Error rate per target must be under 1% after warm-up. Higher indicates throttling or misconfigured routing.
• Sanity ranking: ap-southeast-1 to a US-only endpoint should be slower than us-east-1 to the same endpoint. If not, your probe is caching or your client is misrouted.

Set a simple alert: if p95 TTFT in any region exceeds a threshold (e.g., 2× the baseline from Step 5), page the on-call. This catches provider degradation before users complain.

Common pitfalls when you benchmark llm latency across regions

Ignoring streaming semantics. Non-streaming calls hide TTFT entirely. Always stream.

DNS and connection reuse. Create the client once per region, not per request. Otherwise you pay TLS handshake every call.

System prompt caching. Some providers cache the prefix. If you benchmark with a static prompt, you may measure cache hits that won’t occur in production with user-specific context. Rotate the prompt with a random ID to disable caching, or explicitly forward cache-control hints if your gateway supports it.

Clock skew. Use time.perf_counter() on the probe host; never rely on server timestamps.

Mixed model sizes. Comparing a 8B model to a 70B model on TPS is meaningless. Keep model class fixed per comparison.

Verify your benchmark end to end

You have a working harness when:

1. You can deploy one container image to three regions with a single command.
2. Running it produces JSONL with TTFT, TPS, total, and error per call.
3. Aggregation shows stable p50/p95 across three consecutive runs.
4. The latency ranking by geographic distance matches expectation.

At that point, you have a defensible way to benchmark llm latency across regions and can track regressions as providers change infrastructure or you add new models. Re-run weekly; provider latency is not static.