Why LLM API inference speed varies across providers isn’t a single culprit; it’s the compounding effect of silicon choice, scheduler design, and traffic management. Engineers benchmarking a 70B model on Provider A versus Provider B often see 3–5x latency gaps that have nothing to do with the weights.
The hardware lottery
Providers do not run identical clusters. One may serve a 70B model from H100s with 3.35 TB/s HBM3 bandwidth, while another uses A100s at roughly half that bandwidth. A third might use older A10G or T4 instances to cut cost on cheaper tiers. Memory bandwidth directly gates token generation because each forward pass fetches the full weight set per token.
Compute density matters less than memory-bound decoding for most chat workloads. A provider running fp16 weights on H100 will decode faster than one running the same weights on A100, even at identical batch sizes. AMD MI300X and AWS Inferentia2 change the equation again with different memory hierarchies and kernel maturity.
The kernel ecosystem lags hardware. A provider who invested in custom CUDA or ROCm kernels for continuous batching will outperform one using stock Hugging Face TGI defaults on the same chip. That’s why two providers both claiming “H100” can post different numbers.
Batching and scheduling: the silent throughput killer
Inference servers either statically batch or continuously batch. Static batching waits to collect N requests, then runs them together. This raises throughput but punishes the first request with queue wait time. Continuous batching (used by vLLM, TensorRT-LLM, and others) inserts new requests as older ones finish tokens, keeping GPUs saturated with less tail latency.
A provider’s scheduler policy is invisible in the API but dominates your p95. If Provider X caps batch size at 16 to protect latency, and Provider Y pushes to 64 for throughput, your single low-traffic request may run faster on X even though Y is “faster” in aggregate benchmarks.
What this looks like in code
You can expose scheduler impact by sending concurrent requests and measuring time-to-first-token (TTFT):
import threading, time
from openai import OpenAI
client = OpenAI(base_url="https://api.provider.com/v1", api_key="sk-...")
def fire():
start = time.time()
resp = client.chat.completions.create(
model="mistral-7b",
messages=[{"role":"user","content":"ping"}],
max_tokens=1,
)
print(f"TTFT {time.time()-start:.2f}s")
threads = [threading.Thread(target=fire) for _ in range(20)]
[t.start() for t in threads]; [t.join() for t in threads]If TTFT balloons linearly with concurrency, the provider is queueing rather than batching efficiently.
Quantization and memory layout
Weights served as fp16, int8, fp8, or GPTQ-int4 change both speed and memory footprint. int4 cuts memory bandwidth requirements by ~4x, allowing larger batches or faster decode on the same card. But quantization is not free: dequantization overhead and accuracy loss vary by method.
A provider serving a 70B model in int4 on A100 may match fp16 on H100 for token speed, while sacrificing benchmark scores on rare tasks. You rarely get the quant config in the API response. You infer it from token-per-second anomalies and output drift.
Network path and edge placement
Round-trip latency is part of perceived speed. A provider with a single us-east-1 endpoint forces transcontinental TLS handshakes for EU users. Another with edge PoPs terminates TCP closer, cutting TTFT by 50–100 ms before the model even runs.
Measure the raw transport separately:
curl -w "DNS: %{time_namelookup} Connect: %{time_connect} TTFB: %{time_starttransfer}\n" \
-X POST https://api.provider.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'If time_starttransfer is high but the model is small, the bottleneck is network or load balancer, not inference.
Load shedding and autoscaling behavior
Providers handle overflow differently. Some return 429 immediately; others queue internally with soft timeouts. Under load, a provider may silently drop batch priority, stretching your 200 ms response to 2 s. This variance is why a provider looks fast in isolated tests but degrades in production traffic.
A gateway like n4n.ai provides automatic fallback when a provider is rate-limited or degraded, which hides outages but doesn’t equalize speed; you still pay the retry penalty. The fallback flips to a secondary provider, but TTFT includes the failed attempt plus cold start on the new path.
Measuring speed without lying to yourself
Single-shot curl tests are misleading. You need TTFT and tokens-per-second under representative concurrency. Use the OpenAI streaming interface to capture first byte and subsequent tokens:
from openai import OpenAI
import time
client = OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
start = time.time()
stream = client.chat.completions.create(
model="mixtral-8x7b",
messages=[{"role":"user","content":"Explain TCP fast open in one paragraph"}],
stream=True,
max_tokens=200,
)
first_token = None
tokens = 0
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
if first_token is None:
first_token = time.time()
tokens += 1
ttft = first_token - start
gen_time = time.time() - first_token
tps = tokens / gen_time if gen_time > 0 else 0
print(f"TTFT: {ttft:.2f}s, tokens/s: {tps:.1f}, total tokens: {tokens}")Run this from the same region as your production workers, at multiple hours of the day. Record p50 and p95. A provider with great p50 but terrible p95 will hurt user experience more than one with stable mid-pack latency.
When you have per-token usage metering and can honor client routing directives (as n4n.ai does), you can pin latency-sensitive traffic to a known-fast endpoint and route bulk jobs to cheaper, slower ones. That requires the measurement discipline above; otherwise you’re guessing.
Tradeoffs: speed versus cost and accuracy
Faster inference usually costs the provider more in silicon or engineering time. They pass that on via higher per-token prices or recover it by quantizing aggressively. As a consumer, you trade:
- Latency for throughput when providers batch hard.
- Accuracy for speed when they use int4/int8.
- Availability for raw speed when they shed load instead of queueing.
There is no “best provider” globally. For a coding copilot, TTFT under 300 ms matters more than 100 tok/s. For a bulk summarizer, sustained tok/s at low cost beats TTFT.
Decisive takeaway
Why LLM API inference speed varies is multifactorial: silicon, schedulers, quantization, network, and load policy. Stop trusting provider marketing charts. Stand up a streaming benchmark from your own infrastructure, measure TTFT and tok/s at production-like concurrency, and route on those numbers. Use a gateway to mask degradations, but benchmark the underlying endpoints so you know what you’re actually buying.