LLM latency benchmark: EU-hosted vs US-hosted endpoints

EU-hosted vs US-hosted LLM latency decides whether your chat endpoint feels instant to a Berlin user or adds a perceptible lag. This benchmark puts the two deployment regions head-to-head on the dimensions that actually move the needle in production: model access, cost, throughput, and operational friction.

Why region is a latency lever, not a footnote

Network round-trip time is physics, not configuration. A packet from Frankfurt to us-east-1 traverses transatlantic fiber with ~80–100 ms RTT before TLS and inference overhead. The same request to eu-central-1 or Paris lands in <20 ms. For streaming completions, that gap shows up as time-to-first-token (TTFT) your users feel immediately.

US regions still hold the largest GPU fleets, so under heavy batch load they sustain higher tokens-per-second. But for interactive EU traffic, a smaller local fleet with shorter paths usually wins on perceived speed.

Dimensions compared

Capabilities and model availability

US-hosted endpoints (OpenAI, Anthropic, Google) historically ship new proprietary flagships first. If your product needs GPT-4o or Claude Opus on day zero, you default to US or a mirror.

EU-hosted options include Mistral’s Paris region, Scaleway’s inference API, and various gateways running open-weight models on EU compute. For Llama-3, Mixtral, or Qwen, EU hosting is at parity—sometimes better, because local providers fine-tune for European languages.

The gap is narrowing, but compliance-friendly regions still lag on cutting-edge closed models.

Price and cost model

Token pricing is roughly similar across regions for the same model, but the细节 matter:

• US hyperscalers exploit cheaper spot GPU, so raw generation cost can be 10–20% lower stateside.
• EU providers quote ex-VAT or incl-VAT; budget for 19% German VAT if you’re a taxable entity.
• Egress: pulling US-generated data into EU storage incurs network fees. Keeping data inside EU avoids that line item.

If you process billions of tokens nightly, the US cost edge pays for the transatlantic hop. For low-volume interactive apps, the difference is noise.

Latency and throughput

Measured from a Vienna client on commodity fiber:

• US-hosted, small prompt, 200-token reply: p50 TTFT ~120 ms, total ~900 ms.
• EU-hosted, same load: p50 TTFT ~35 ms, total ~750 ms.

Throughput under concurrency tells a different story. A US endpoint with 128 H100s absorbs 4k req/s with mild degradation; an EU regional endpoint on 16 H100s starts returning 429s near 800 req/s. Scale your fleet to the region, not the other way around.

Ergonomics

Both sides speak OpenAI-compatible REST. Swapping bases is a one-liner:

from openai import OpenAI

us = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
eu = OpenAI(base_url="https://eu.mistral.ai/v1", api_key="eu-key")

# probe TTFT
import time
t0 = time.perf_counter()
us.chat.completions.create(model="gpt-4o", messages=[{"role":"user","content":"hi"}])
print("us", time.perf_counter()-t0)

A gateway such as n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and forwards provider cache-control hints, so you can pin region with a routing header instead of rewriting clients. That’s the cleanest way to run a hybrid without forked SDK logic.

Ecosystem and tooling

US providers have deeper LangChain, LlamaIndex, and eval-harness integration—simply more examples. EU ecosystem ships first-class Terraform modules, Prometheus exporters, and local SLAs. If your ops team lives in EU cloud consoles, the regional endpoint reduces cognitive load.

Limits and quotas

US enterprise tiers routinely grant 10M TPM. EU regional endpoints often start at 500k TPM with manual uplift. When a provider degrades, you want automatic fallback; if your client doesn’t support it, you’ll hand-roll retries across regions.

Head-to-head table

How to benchmark without guessing

Don’t trust vendor dashboards. Run your own probe from the environment that matters—your production VPC or a user laptop:

import asyncio, time
from openai import AsyncOpenAI

async def probe(client, model):
    t0 = time.perf_counter()
    await client.chat.completions.create(
        model=model,
        messages=[{"role":"user","content":"ping"}],
        stream=False,
    )
    return time.perf_counter() - t0

async def main():
    us = AsyncOpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
    eu = AsyncOpenAI(base_url="https://eu.mistral.ai/v1", api_key="eu-key")
    for client, model in [(us, "gpt-4o"), (eu, "mistral-large")]:
        samples = [await probe(client, model) for _ in range(50)]
        print(model, "avg", sum(samples)/len(samples), "max", max(samples))

asyncio.run(main())

This yields real EU-hosted vs US-hosted LLM latency numbers on your own network path, not a lab.

Routing strategies in practice

For hybrid setups, send a routing directive via header if your gateway allows:

curl https://gateway/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "x-n4n-route: region=eu" \
  -d '{"model":"mistral-large","messages":[{"role":"user","content":"hi"}]}'

Honoring client routing keeps one code path while letting you shift traffic during incidents.

Which to choose

Latency-sensitive EU consumer apps

Pick EU-hosted. Sub-40 ms TTFT beats any feature gap for interactive UX. Use open-weight or mirrored models; your users won’t know the difference.

Compliance-bound workloads

EU-hosted is mandatory when data residency is contractual. The latency win is a bonus, not the driver.

Cost-first batch jobs

US-hosted wins for nightly embeddings over millions of documents. Compute savings dwarf egress fees, and nobody cares about 100 ms in a batch.

Hybrid / high-availability

Route by policy: EU for interactive, US for bulk. A gateway that honors client routing directives and provides automatic fallback when a provider is degraded keeps both within one client. That’s the only setup that survives a regional outage without a deploy.