Why LLM latency differs by region and how to fix it

Understanding why LLM latency varies by region starts with provider infrastructure, not just the speed of light. A request from São Paulo to a model hosted only in Virginia incurs transit, TLS, and queueing costs that no client-side tweak can erase. This guide gives an ordered path to measure those gaps and engineer around them with routing, pinning, and fallback.

What actually causes regional latency gaps

Physical distance is the obvious factor, but it’s rarely dominant for sub-100ms links. The bigger variables are where the provider placed the model weights, how they shard batches across GPUs, and whether the endpoint runs a cold model instance.

Most inference providers replicate popular models to a few regions. If your user is outside those, the gateway proxies the request, adding a hop. Some providers also apply regional rate limits, so a busy region queues requests even when compute is free elsewhere.

Quantization and serving stack differences matter too. A region running FP8 might return tokens materially faster than one on BF16, even at identical hardware. You cannot assume homogeneity.

Measure before you optimize

You cannot fix what you haven’t timed. Run synthetic requests from representative client locations using a minimal script. The data you collect is the raw evidence for why LLM latency varies by region and where your money is leaking.

Below is a Python snippet that fires a chat completion to an OpenAI-compatible endpoint and records time-to-first-token (TTFT) and total latency.

import asyncio, time, httpx

async def probe(base_url: str, region_hint: str | None):
    headers = {"Authorization": "Bearer $API_KEY"}
    if region_hint:
        headers["x-preferred-region"] = region_hint
    payload = {
        "model": "openai/gpt-4o-mini",
        "messages": [{"role": "user", "content": "ping"}],
        "max_tokens": 16,
        "stream": True,
    }
    start = time.perf_counter()
    async with httpx.AsyncClient(timeout=30) as c:
        async with c.stream("POST", f"{base_url}/chat/completions", json=payload, headers=headers) as r:
            ttft = None
            async for chunk in r.aiter_bytes():
                if ttft is None:
                    ttft = time.perf_counter() - start
                if b"done" in chunk:
                    break
    total = time.perf_counter() - start
    return ttft, total

async def main():
    for region in ["us-east", "eu-west", "ap-south"]:
        ttft, total = await probe("https://api.example.com/v1", region)
        print(f"{region}: TTFT={ttft*1000:.0f}ms total={total*1000:.0f}ms")

asyncio.run(main())

Run this from VMs in each target region. Look at p50 and p99, not averages. A region with great p50 but terrible p99 will burn your SLA during traffic spikes.

Common pitfall: measuring from your laptop. Home broadband introduces jitter that masks provider differences. Use cloud instances.

Pick a deployment topology

You have three basic options:

1. Single global anycast endpoint – simplest. The network routes you to nearest POP, but the model might still live far away.
2. Region-specific endpoints – you call https://eu-west.api.example.com/v1 explicitly. More control, more code.
3. Gateway with edge routing – a layer that honors client routing directives and forwards requests to the closest healthy region. Some gateways like n4n.ai honor client routing directives and forward provider cache-control hints, which preserves prefix caches across edges.

Tradeoff: anycast hides complexity but gives zero visibility. Explicit endpoints force you to build region selection logic. A gateway sits in the middle, adding a small proxy cost but enabling automatic fallback.

Use client-side region pinning

If you go with explicit endpoints or a gateway that supports headers, pin traffic based on user geolocation. Don’t guess—use the request origin or a config map.

from fastapi import Request, FastAPI
from openai import OpenAI

app = FastAPI()

REGION_MAP = {"US": "us-east", "EU": "eu-west", "ASIA": "ap-southeast"}

@app.post("/chat")
async def chat(req: Request):
    user_region = req.headers.get("x-user-geo", "US")
    preferred = REGION_MAP.get(user_region, "us-east")
    client = OpenAI(
        base_url="https://api.example.com/v1",
        api_key="...",
        default_headers={"x-preferred-region": preferred}
    )
    resp = client.chat.completions.create(
        model="anthropic/claude-3.5-sonnet",
        messages=[{"role": "user", "content": "Hello"}]
    )
    return resp.model_dump()

Pitfall: pinning too rigidly causes hotspots. If eu-west degrades, you must detect and shift. Build a circuit breaker.

Implement fallback and degradation handling

Regional outages and rate limits happen weekly for major providers. Your code should retry with region rotation on 429 or 503.

import random, time
from openai import OpenAI, APIStatusError

REGIONS = ["us-east", "eu-west", "ap-south"]

def call_with_fallback(prompt: str):
    regions = random.sample(REGIONS, len(REGIONS))
    for region in regions:
        try:
            client = OpenAI(
                base_url="https://api.example.com/v1",
                api_key="...",
                default_headers={"x-preferred-region": region}
            )
            return client.chat.completions.create(
                model="openai/gpt-4o",
                messages=[{"role": "user", "content": prompt}]
            )
        except APIStatusError as e:
            if e.status_code in (429, 503):
                time.sleep(0.5)
                continue
            raise
    raise RuntimeError("All regions failed")

Automatic fallback when a provider is rate-limited or degraded is non-negotiable for production. If you use a gateway, ensure it does this for you; otherwise you own the logic.

Cache where it counts

Prompt caching slashes TTFT for repeated system prompts. But caches are usually regional. If you pin users to one region, their prefix cache stays warm. If you bounce them around, you pay cold start each time.

Forward provider cache-control hints. When your gateway passes cache-control: max-age=... to the provider, you keep the benefit even through a proxy. Without that, the gateway may strip it and you lose a large part of the speedup on long system prompts.

Tradeoff: caching ties you to a region, reducing fallback options. Decide based on prompt size—small prompts don’t justify pinning.

Common pitfalls and tradeoffs

• Connection reuse: Not reusing clients via httpx or OpenAI singletons adds handshake overhead per call. Always reuse clients.
• Synchronous code: Blocking I/O in a web server magnifies latency under load. Use async.
• Ignoring p99: A region can look fine at p50 but drop a slice of requests at multi-second tails. Measure tails.
• Over-pinning: Rigid region locks prevent fallback. Use soft preferences with health checks.

Actionable checklist

1. Profile TTFT and total latency from each target region using the script above.
2. Map providers’ actual model locations—don’t trust docs; test.
3. Choose topology: explicit endpoints for control, gateway for automation.
4. Implement region pinning via headers or endpoint selection, with geo lookup.
5. Add fallback loop for 429/503 with jittered backoff.
6. Enable prompt caching and verify cache-control headers survive your stack.
7. Monitor p99 per region; alert on degradation.

Following this path turns why LLM latency varies by region from a mystery into a tuned parameter set. The fixes are engineering, not magic.