Why tokens per second varies by provider is not a mystery of model weights—it is a systems problem. The same 70B parameter model served from a Virginia datacenter versus one in Frankfurt can differ by 2x in sustained throughput because of GPU SKU differences, queue depth, and interconnect topology. Engineers benchmarking LLM endpoints must separate model efficiency from infrastructure allocation before trusting any published ranking.
The hardware floor: GPU SKUs and interconnects
Token throughput during decode is dominated by memory bandwidth, not raw FLOPS. Each generated token requires a full weight read from HBM. A GPU with 2 TB/s bandwidth will sustain roughly double the tokens per second of one with 1 TB/s, assuming identical batch behavior.
Silicon differences that matter
H100’s HBM3 and FP8 tensor cores reduce the time per forward pass for quantized models. A100-40GB clusters are still widespread because of amortized cost, but they penalize large context windows and limit batch size. MI300X offers higher bandwidth again, yet software stack maturity varies by provider, which itself shifts observed TPS.
NVLink vs PCIe spine
Multi-GPU inference for a single model shard relies on all-reduce across layers. A node with an NVLink spine completes layer sync in microseconds; a node daisy-chained on PCIe 4.0 adds milliseconds per step. If a provider packs GPUs into a low-cost PCIe topology to maximize density, per-token latency climbs even when paper FLOPS look similar.
Batching and scheduling: where TPS is actually made
Providers do not serve one request at a time. Continuous batching aggregates many streams into a single CUDA graph execution, and the scheduler implementation is the single biggest differentiator between vendors.
Continuous batching internals
The scheduler pads sequences to the longest active request. If your 128-token prompt shares a batch with a 32k context query, your tokens wait behind its KV reads. Some providers use vLLM-style paged attention; others use TensorRT-LLM or custom kernels. The paged approach reduces memory fragmentation and raises effective batch occupancy, directly lifting tokens per second.
Max batch size tradeoffs
A larger batch raises GPU utilization and tokens/sec, but also increases time-to-first-token (TTFT). A provider optimizing for chatbot feel will throttle batch size; one selling bulk summarization will max it. The same model, two TPS profiles. This is a primary reason why tokens per second varies by provider even within one region.
{
"model": "mistral-7b-instruct",
"max_batch_size": 64,
"scheduling_policy": "throughput"
}That config snippet is illustrative of what a provider’s internal scheduler might accept. You do not control it directly, but your request pattern interacts with it.
Regional capacity allocation and load shedding
Why tokens per second varies by provider and region becomes obvious once you map where silicon is deployed and how traffic is steered.
Newest hardware lands in flagship regions
Cloud regions like us-east-1 or eu-west-1 receive H100 allocations first. Secondary regions run older stock or smaller clusters because of procurement cycles and power constraints. During peak hours, providers shed load by routing to distant regions, inflating network RTT and reducing effective streaming TPS.
Degraded modes and fallback
When a zone is rate-limited, gateways fail over. An inference gateway such as n4n.ai honors client routing directives and forwards provider cache-control hints, so you can pin a region to stabilize throughput or allow fallback when degraded. Without pinning, your TPS measurement becomes a moving average across heterogeneous hardware and scheduler versions.
Network egress and TTFT vs TPS
Tokens per second is measured at the client. The bytes travel, and the path quality shapes the number you record.
Streaming over long haul
A model generating 80 TPS on the host may deliver 60 TPS to a client 200 ms away because of TCP congestion and chunking. HTTP/2 flow control can stall if your consumer is slower than the producer. QUIC or dedicated peering reduces this, but not every provider exposes it in every region.
Measurement pitfalls
If you count tokens from the SSE stream but include the TTFT in the denominator, you punish the provider for network setup. Correct TPS isolates generation phase:
start_gen = None
tokens = 0
for chunk in stream:
if chunk.choices[0].delta.content:
if start_gen is None:
start_gen = time.time()
tokens += 1
gen_tps = tokens / (time.time() - start_gen)That isolates generation from connection overhead and reveals the real server-side rate.
How to measure TPS that means something
Publishable comparisons require fixed variables. Treat a benchmark like a scientific assay: control the substrate.
Pin the region and model revision
Always specify model@revision and a region header. A request without routing hints gets round-robined across zones and SKUs.
curl https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "X-Route-Region: us-east-1" \
-d '{"model":"llama-3-8b","stream":true,"messages":[{"role":"user","content":"loop 300 words"}]}'Run at least 20 trials per configuration. Report median and p90, not mean, because batch neighbors introduce skew.
Account for cache hits
Provider cache-control hints can skip prompt recompute. A cached prefix generates faster subsequent tokens. Forward the hints or your benchmark conflates cold and warm paths, masking a major throughput lever.
client.chat.completions.create(
model="llama-3-8b",
messages=[{"role":"user","content":"large cached doc"}],
extra_body={"cache_control": {"type":"ephemeral"}}
)Tradeoffs: optimize for cost, latency, or throughput
You cannot max all three. The infrastructure that delivers highest TPS often costs more per token and worsens TTFT.
Multi-region routing
Pinning to a premium region costs more per token but yields predictable TPS. Allowing fallback cuts cost but introduces variance. For batch jobs, variance is fine; for interactive agents, it breaks UX. A nightly ETL pipeline tolerates a 30% TPS swing; a coding copilot does not.
Gateway-level controls
Per-token metering lets you attribute TPS drops to a specific provider or zone. When a region degrades, automatic fallback preserves availability at the expense of throughput. Decide based on workload class, not vendor marketing.
Takeaway
Why tokens per second varies by provider is answered by hardware heterogeneity, scheduler policy, and regional capacity—not model architecture alone. Measure generation-phase TPS with region pinned and cache state controlled, then treat provider averages as priors rather than guarantees. For production, choose a gateway that exposes routing directives and per-token metering so you can trade cost for predictability with eyes open.