Most published network latency llm benchmark results quietly include the cost of traversing multiple network hops between the client and the inference provider. That transport overhead routinely dwarfs the differences between models under test, yet it rarely appears in the methodology section. If you measure a model’s time-to-first-token from a laptop in Berlin against a US-east endpoint, you are mostly measuring the Atlantic.
The hidden variable in every LLM timing
Engineers benchmark LLMs to compare compute efficiency: tokens per second, inter-token latency, cold-start penalty. But the wire is not free. Every additional router, proxy, or load balancer inserts queuing delay, TLS termination, and retransmission risk.
A typical request path from a developer machine looks like this: your process → local NAT → ISP → transit provider → edge PoP → gateway → provider load balancer → actual GPU worker. That is six or more logical hops. Each adds 0.5–20 ms when healthy, and seconds if congested or misrouted.
Even inside a single cloud region, cross-availability-zone traffic traverses leaf and spine switches. A call from a container in us-east-1a to a model worker in us-east-1c pays at least one intra-region router tax that does not exist for colocated pods. Ignore this and your comparison between two providers in the same region is still polluted.
What a “hop” actually costs
TCP/TLS handshake and connection reuse
A naive benchmark opens a new HTTPS connection per request. That costs one TCP three-way handshake (one round-trip) plus a TLS 1.3 handshake. With session resumption, TLS adds another RTT; without, it adds two. At 80 ms round-trip time, that is 160–240 ms of dead time before the first byte of your prompt leaves the client.
import time, httpx
async def timed_request(client, url, json):
start = time.perf_counter()
r = await client.post(url, json=json)
return time.perf_counter() - start
# Without connection pooling, each call pays handshake cost
async def naive_bench(url, n=10):
totals = []
for _ in range(n):
async with httpx.AsyncClient() as c: # new client, new conn
totals.append(await timed_request(c, url, {"prompt":"hi"}))
return totalsReusing a client with keep-alive collapses that to near zero. Most serious network latency llm benchmark work must state connection reuse policy, because the difference between cold and warm connection is often larger than the gap between two models.
HTTP/2 multiplexing helps but only if the SDK actually uses it. Many Python and JS SDKs default to one request per connection or open a new pool per process. Verify with ss -tnp or a proxy log.
Geographic routing and anycast
Providers often use anycast DNS and edge termination. Your resolver decides which continent answers. Two consecutive runs can hit Frankfurt or Virginia based on BGP weather. That variance is not model behavior.
Traceroute is informative but misleading: many backbone routers deprioritize ICMP, so reported latency at hop 9 may be 30 ms while the real forwarding path is 2 ms. Use TCP-based measurement instead.
mtr --report --tcp --port 443 api.example.comIf you see 14 hops with 30% loss at a transit provider, your latency numbers are noise. Run from multiple vantage points.
Why most benchmarks measure the wrong thing
Client-side measurement bias
Measuring from the caller’s machine captures local CPU scheduling, DNS latency, and wifi jitter. A 2015 MacBook with a sleeping NIC will report worse “model latency” than a wired server in a rack. The client clock also includes time spent in SDK retry logic. OpenAI-compatible SDKs often retry on 429 with exponential backoff; that backoff is infrastructure, not inference.
Worse, client-side timers start before DNS resolution and end after the last socket read, blending network idle with compute. If the provider streams tokens slowly due to network congestion, you attribute that to the model’s generation speed.
Server-side vs gateway-mediated
If you benchmark directly against a provider, you omit the gateway hop that production uses. If you benchmark through a gateway, you include it. A network latency llm benchmark that silently switches between these setups is invalid for comparison.
Consider a gateway that adds automatic fallback when a provider is rate-limited. The fallback may add 200 ms but save a failed run. Your benchmark either must simulate degradation or exclude it explicitly. Gateways also perform request validation, token counting, and routing logic that consume CPU on the critical path.
A minimal measurement harness
Isolate model time by measuring from a machine in the same region as the endpoint, with warmed connections. The cleanest approach: run a tiny client on a cloud instance colocated with the provider, reuse a single HTTP client, and stream.
import asyncio, time, httpx, statistics
URL = "https://api.example.com/v1/chat/completions"
PAYLOAD = {
"model": "test-model",
"messages": [{"role":"user","content":"Write a haiku."}],
"stream": True,
}
async def stream_first_token():
async with httpx.AsyncClient(timeout=30) as client:
start = time.perf_counter()
async with client.stream("POST", URL, json=PAYLOAD) as r:
async for chunk in r.aiter_text():
if chunk:
return time.perf_counter() - start
return None
async def main(runs=20):
await stream_first_token() # warm
samples = [await stream_first_token() for _ in range(runs)]
print(f"p50={statistics.median(samples)*1000:.1f}ms")
asyncio.run(main())This still includes one hop inside the region, but that is constant and small. For inter-token latency, parse the stream and timestamp each delta:
async def stream_inter_token():
async with httpx.AsyncClient() as client:
async with client.stream("POST", URL, json=PAYLOAD) as r:
last = time.perf_counter()
deltas = []
async for chunk in r.aiter_text():
now = time.perf_counter()
if chunk:
deltas.append(now - last)
last = now
return deltasWhat to log
Always emit metadata so the result is reproducible:
{
"client_region": "us-east-1",
"endpoint_region": "us-east-1",
"tls_resumed": true,
"retries": 0,
"gateway_hop": false,
"sample_size": 100,
"transport": "http2"
}Without this, a network latency llm benchmark is just an anecdote.
Tradeoffs of colocating vs remote
Colocating the benchmark client with the model removes network variance but stops representing real users. A mobile app in São Paulo will never see your us-east colo numbers. Remote measurement reflects user experience but needs statistical rigor: 100+ samples, trimmed mean, and outlier exclusion.
Tradeoff summary:
- Colocated: low noise, good for model-vs-model diffs, useless for SLOs.
- Remote from target market: high variance, requires large n, reflects reality.
- Synthetic backbone monitor: middle ground, catches provider regressions but misses last-mile.
The network latency llm benchmark that matters for product is the remote one, but you must annotate it as such. Use colocated runs only to normalize.
Gateway considerations
Production systems rarely call providers directly. An OpenAI-compatible gateway like n4n.ai honors client routing directives and forwards provider cache-control hints, but the extra hop still adds measurable latency. If your benchmark runs through such a gateway, subtract the gateway’s own round-trip baseline measured against a dummy endpoint, or compare like-for-like.
Automatic fallback when a provider is degraded is a feature, not a bug, but it widens latency distribution. Capture p95 and p99, not just mean. Per-token metering at the gateway adds negligible CPU but can introduce a small serialization step; include it in the documented path.
Decisive takeaway
Network hops are not a constant you can ignore; they are the dominant variable in most latency measurements outside a datacenter. Publish your network topology, reuse connections, run enough samples, and separate transport cost from compute cost. Only then does a network latency llm benchmark tell you which model is actually faster rather than which continent is closer.
If you take one action: re-run your last benchmark with mtr open and a warmed client. The numbers will change, and now they will mean something.