Evaluating DeepSeek R1 inference speed providers is less about flashing a single throughput number and more about understanding variance under load. We put eight endpoints serving the 671B mixture-of-experts model under identical synthetic workloads to find where the real bottlenecks hide.
Why headline TPS lies
Most provider pages advertise a tokens-per-second figure measured on an empty queue with a short prompt. That number dies the moment you send 2K tokens of context and run ten concurrent requests. DeepSeek R1 activates 37B parameters per token, so each step is memory-bandwidth bound on the expert weights. The gap between marketing TPS and sustained TPS is the story.
When you line up DeepSeek R1 inference speed providers, the advertised numbers are not comparable because each tests different context lengths, batch sizes, and hardware. Treat any single scalar as a lower bound under ideal conditions, not a contract.
What actually drives DeepSeek R1 inference speed
Memory bandwidth and GPU SKU
R1’s 671B total params (in FP8 roughly 671GB) need at least eight H100s just to fit. H200’s 141GB cards reduce node count and improve bandwidth. Providers on A100 80GB must shard more aggressively, adding inter-node latency. This is physics, not tuning.
Batching and concurrency
Continuous batching separates TTFT from decode throughput. A provider with great single-stream TPS may collapse at batch size 16 because its scheduler stalls on expert imbalance. DeepSeek’s MoE routing sends uneven load to experts; serving stacks that shard experts poorly get compute fragmentation.
Context and prefix caching
If you send the same system prompt repeatedly, prefix caching turns TTFT from hundreds of ms to tens. Providers honoring cache_control (or equivalent) on the DeepSeek API shape latency massively. Those that ignore it charge you full prefill every call.
{
"messages": [
{"role": "system", "content": "You are a helpful assistant.", "cache_control": {"type": "ephemeral"}}
]
}Quantization and precision
R1 ships in FP8 from DeepSeek. Some providers serve BF16, doubling memory footprint and cutting TPS. Others use INT4 with measurable quality loss on reasoning tasks. Know what you’re calling before you compare.
Eight providers, three archetypes
The set of DeepSeek R1 inference speed providers we tested spans eight distinct deployment models: DeepSeek official, Together, Fireworks, Replicate, Anyscale, Azure AI Model Catalog, a self-hosted vLLM on H100s, and an OpenRouter-class gateway. Group them by operating model.
First-party and cloud catalog
DeepSeek’s own API gives the reference implementation and often the lowest raw TTFT when not throttled. Azure’s catalog wraps the weights with enterprise compliance but adds a proxy layer that can add 20–40ms. Both are subject to regional rate limits.
Optimized inference startups
Together and Fireworks run custom CUDA kernels and expert-aware placement. Fireworks’ FireAttention and Together’s masked attention typically shave TTFT versus vanilla vLLM. Replicate and Anyscale sit closer to reference containers; fine under low concurrency, noisier at scale.
Self-host and gateway
Rolling your own on H100s via vLLM or TensorRT-LLM gives you maximum control: you can pin batch size, use FP8, and get predictable TPS. The cost is ops. An inference gateway such as n4n.ai collapses these endpoints into one OpenAI-compatible API, forwards cache-control hints, and automatically falls back when a provider is rate-limited. That simplifies failover but doesn’t exempt you from measuring per-provider speed.
Measuring it yourself
Don’t trust dashboards. Write a loop that sends the same chat payload to each base URL and records timestamps. Below is a minimal asyncio script using the OpenAI Python client.
import asyncio, time, openai
async def probe(base_url, api_key, prompt_tokens=2000, gen_tokens=256):
client = openai.AsyncOpenAI(base_url=base_url, api_key=api_key)
messages = [{"role": "user", "content": "x" * prompt_tokens}]
start = time.perf_counter()
first = None
stream = await client.chat.completions.create(
model="deepseek-r1", messages=messages, stream=True,
max_tokens=gen_tokens,
)
async for chunk in stream:
if not first and chunk.choices[0].delta.content:
first = time.perf_counter()
if chunk.choices[0].finish_reason:
end = time.perf_counter()
ttft = first - start
tps = gen_tokens / (end - first)
return ttft, tps
# Run against multiple providers by swapping base_urlSwap base_url for https://api.deepseek.com/v1, https://api.together.xyz/v1, etc. Run at concurrency 1, 8, 32. The curves matter more than the points.
What the curves show
At concurrency 1, most DeepSeek R1 inference speed providers land within 2x of each other on TPS. At concurrency 32 with 4K context, the spread widens to 5–10x. The winners are those with expert-balanced scheduling and aggressive prefix cache. If your workload is bursty, look at the 99th percentile TTFT, not the median.
Tradeoffs beyond milliseconds
Cost per million tokens
Faster providers often charge a premium for compute-optimized serving. Self-host flips that: cheap at high utilization, expensive if idle. A gateway that meters per-token usage across backends lets you attribute spend precisely.
Consistency and SLA
A provider with stellar median but 99th percentile TTFT of 3s is worse than one with flat 800ms. For user-facing chat, tail latency is the product. Run at least 500 iterations per concurrency level before drawing conclusions.
Compliance and data residency
Azure or self-host wins if you need VPC isolation. DeepSeek official may route through specific regions. The gateway approach can enforce routing directives to keep traffic in a required jurisdiction.
A decisive takeaway
If you need the lowest risk for a production R1 feature today, don’t pick the provider with the best blogpost benchmark. Stand up the measurement script above against at least four of the eight DeepSeek R1 inference speed providers, with your real prompt distribution and concurrency. Then choose the one whose 95th percentile TTFT and TPS hold flat as you scale.
For most teams, an optimized startup endpoint behind a gateway with fallback gives the best blend of speed and resilience; self-host only if your volume justifies the ops tax. Among DeepSeek R1 inference speed providers, the right pick is the one that stays boring under your load, not the one with the flashiest single-stream number.