Any published DeepSeek V3 throughput benchmark hides more than it reveals. Provider-hosted instances of this 671B mixture-of-experts model differ in GPU topology, serving framework, and batching strategy, so a single number for tokens/sec is only meaningful for the exact load that produced it.
Why provider throughput varies
DeepSeek V3’s 671B total parameters with 37B active per token mean the weight footprint is large but the compute per token is moderate. The 256 experts are distributed across layers; routing decisions happen per token. Serving systems must either replicate the router or synchronize across nodes, which introduces latency that never appears in single-GPU tests.
Model architecture constraints
The MoE layer forces a tradeoff between expert parallelism and communication overhead. A provider can shard experts across GPUs, but all-to-all communication between token routing and expert compute becomes the bottleneck at high batch sizes. Some keep all experts on one node with NVLink; others spread them across InfiniBand-connected nodes, trading latency for capacity. A DeepSeek V3 throughput benchmark run on a single-node config will not generalize to a multi-node one.
Serving stack differences
vLLM, TensorRT-LLM, and SGLang implement continuous batching differently. vLLM’s paged KV cache is flexible but adds overhead for very long contexts. TensorRT-LLM often wins on raw decode speed for fixed batch shapes but is slower to adapt to variable sequences. SGLang’s radix attention can reuse common prefixes across requests, boosting effective throughput for shared system prompts—if your benchmark omits a shared prefix, you miss that gain. The same model on the same silicon can post wildly different numbers depending on the framework.
Hardware and parallelism
FP8 quantization is supported by DeepSeek V3 weights; H100/H200 accelerate it. A provider serving BF16 uses 2x memory and halves achievable batch size per GPU. PCIe vs SXM variants change interconnect bandwidth, which matters when tensor parallelism spans cards. Common layouts use TP=8 on a single node for attention and EP=16 across two nodes for experts. Without InfiniBand, expert shuffles stall.
Defining the metrics
Before comparing numbers, agree on what you measure.
Time to first token (TTFT)
Wall-clock from request send to first byte of output. Dominated by prefill and scheduler queueing.
Decode tokens per second per stream
Output tokens divided by decode duration. This is what users feel during streaming.
Aggregate cluster throughput
Total output tokens across all concurrent streams per second. This is what the provider’s billing and capacity planning care about.
A DeepSeek V3 throughput benchmark at concurrency 1 measures latency, not throughput. At concurrency 64, you measure the provider’s ability to pack sequences.
What a benchmark actually measures
Throughput is not a single metric. Separate prefill from decode.
Prefill vs decode
Prefill is compute-bound and benefits from large batch matrices. Decode is memory-bandwidth-bound and suffers as batch size grows. A provider optimized for coding assistants (short prompts, long outputs) shows different numbers than one optimized for RAG (long prompts, short outputs).
Concurrency and batch size
Sustained throughput scales with concurrent requests until KV cache or scheduler saturates. Beyond that, per-stream TPS collapses while aggregate may still climb. You need both curves.
Context length effects
Longer contexts consume KV cache, reducing room for batching. If your real traffic mixes 2K and 32K contexts, a provider that looks fast on a 1K-context synthetic test may degrade sharply. Always benchmark with your p50 and p95 prompt lengths.
How to run your own DeepSeek V3 throughput benchmark
Don’t trust vendor charts. Write a script that mirrors your traffic shape.
Minimal streaming client
from openai import OpenAI
import time
client = OpenAI(base_url="https://api.provider-x.com/v1", api_key="sk-...")
prompt = "Explain the tradeoffs of MoE inference serving." * 50 # ~300 tokens
start = time.time()
first_token = None
output_tokens = 0
stream = client.chat.completions.create(
model="deepseek/deepseek-v3",
messages=[{"role": "user", "content": prompt}],
stream=True,
max_tokens=512,
temperature=0,
)
for chunk in stream:
if chunk.choices[0].delta.content:
if first_token is None:
first_token = time.time()
output_tokens += 1
end = time.time()
ttft = first_token - start
decode_tps = output_tokens / (end - first_token)
print(f"TTFT: {ttft:.2f}s, Output tokens: {output_tokens}, Decode TPS: {decode_tps:.1f}")Concurrency harness
import concurrent.futures as cf
def worker(_):
# same streaming call as above, return (decode_tps, ttft)
return decode_tps, ttft
with cf.ThreadPoolExecutor(max_workers=32) as ex:
results = list(ex.map(worker, range(32)))Run this for fixed durations and capture percentiles.
Controlling for variables
Fix prompt length, output length, and temperature. Use max_tokens to cap generation. Record p50/p95 TTFT and tokens/sec per stream, plus aggregate cluster throughput. If a provider supports cache-control headers, send them to mimic production reuse.
{
"model": "deepseek/deepseek-v3",
"messages": [{"role": "user", "content": "..."}],
"max_tokens": 512,
"temperature": 0,
"stream": true,
"headers": {"x-cache-control": "prefix-cache"}
}Provider comparison: qualitative observations
Without naming vendors, the field splits into three buckets.
Cloud GPU aggregators
These resell spare H100/A100 capacity. Throughput is inconsistent: you may get a bare-metal instance with good NVLink, or a crowded multi-tenant node. A DeepSeek V3 throughput benchmark here swings between runs because the underlying instance type changes.
Dedicated inference platforms
They compile the model for a fixed serving stack and publish SLAs. Expect higher and more stable decode TPS, but less flexibility on context window or quantization. Custom CUDA graphs reduce kernel launch overhead but lock the batch shape.
Gateways and fallback
A gateway such as n4n.ai exposes one OpenAI-compatible endpoint for 240+ models and forwards provider cache-control hints, so you can point the same benchmark script at different backends by changing a base URL. Its automatic fallback also keeps your load test running when a provider throttles, which avoids skewed results from partial outages. Per-token usage metering lets you attribute cost per provider without instrumenting each client.
Tradeoffs: latency, cost, stability
High throughput often means large batches, which raises per-request TTFT. If your app is interactive, a provider averaging high tokens/sec but 2s TTFT is worse than one at half the rate with 200ms TTFT. Cost compounds: per-token pricing varies, and a provider that silently truncates long contexts forces retries.
Stability is the silent killer. A DeepSeek V3 throughput benchmark that runs for five minutes may look great; run it for six hours and watch for scheduler thrash or OOM kills when the provider’s other tenants spike. A gateway with fallback masks transient degradation but also hides it—log which backend served each request.
Takeaway
Pick a provider based on your own measured p95 decode throughput under representative concurrency, not a headline DeepSeek V3 throughput benchmark. Use a thin gateway to swap backends without code changes, cap your context, and weight cost-per-1K-output-tokens against latency before committing. The only number that matters is the one your production traffic produces.