n4nAI
Open console →
Benchmarks & performanceAnalysisBatch Inference Throughput Benchmarks

Batch inference throughput benchmark for DeepSeek V3

Analysis of DeepSeek V3 throughput benchmark: MoE batching behavior, KV cache limits, and how to measure real-world inference performance.

n4n Team5 min read1,196 words

Audio narration

Coming soon — every post will get a voice note here.

The DeepSeek V3 throughput benchmark is the first number most teams demand before they commit GPU budget to this 671B-parameter Mixture-of-Experts model. But raw tokens-per-second figures from vendor slides hide the variables that actually determine whether your batch pipeline stalls or scales.

The MoE advantage is real but conditional

DeepSeek V3 ships 671B parameters but activates only ~37B per token via fine-grained expert routing. That architectural choice is the biggest reason a DeepSeek V3 throughput benchmark looks nothing like a dense Llama-405B run on the same H100 node. You load the full weight set into HBM once, then select expert subsets per token.

The multi-head latent attention (MLA) layer compounds the win. By compressing the KV cache into low-rank latent vectors, DeepSeek V3 cuts per-token KV memory by roughly an order of magnitude versus standard MHA. That directly raises the number of concurrent sequences you can hold before hitting the --max-num-seqs wall in vLLM or the equivalent in SGLang.

Building a benchmark that isn’t lying to you

Too many published numbers fix the prompt at 32 tokens and output at 128, then declare victory. Real workloads have long system prompts, retrieval prefixes, and variable completions. Your DeepSeek V3 throughput benchmark must mirror your production distribution.

A minimal defensible setup:

  • 8× H100 SXM5 (or H800) per node, NVLink connected.
  • vLLM 0.6.x or SGLang latest, tensor parallel size 8.
  • Enable prefix caching. MLA makes this especially effective.
  • Sweep --max-num-seqs from 1 to 256 in powers of two.
  • Hold input tokens at your p50 (e.g., 2K) and output at p90 (e.g., 1K).

Launch:

python -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 8 \
  --enable-prefix-caching \
  --max-num-seqs 64 \
  --max-model-len 16384

Drive it with concurrent async clients. The point is to saturate the scheduler, not measure a single request:

import asyncio, openai

async def complete(client, req):
    return await client.chat.completions.create(**req)

async def main(n):
    client = openai.AsyncOpenAI(
        base_url="http://localhost:8000/v1", api_key="EMPTY")
    reqs = [{
        "model": "deepseek-ai/DeepSeek-V3",
        "messages": [{"role": "system", "content": PREFIX},
                     {"role": "user", "content": f"Question {i}"}],
        "max_tokens": 1024,
    } for i in range(n)]
    await asyncio.gather(*[complete(client, r) for r in reqs])

asyncio.run(main(64))

Measure aggregate output tokens divided by wall-clock from first request start to last finish. That is your batch throughput.

What the curve actually looks like

I won’t quote specific tokens/sec because they shift with driver versions and cooling. The shape is consistent:

  • Batch 1 to 8: Near-linear gain. Compute units were starved; now they have work.
  • Batch 8 to 64: Sublinear but strong. KV cache grows, but MLA keeps it manageable.
  • Batch 64 to 256: Diminishing returns. You are memory-bandwidth and scheduler-bound. Preemption kicks in if max model len is high.

The DeepSeek V3 throughput benchmark diverges from dense models in that middle region. A dense 70B might plateau at batch 32 on the same box; MoE + MLA pushes the knee further out.

Sequence length is the hidden tax

Double input length and you roughly double KV footprint per sequence. Even with MLA, a 32K context batch hits --max-num-seqs limits far earlier than a 2K batch. A benchmark using short prompts over-provisions concurrency for production long-context jobs.

Scheduling: continuous batching is non-negotiable

Static batching (padding N requests to same length) wastes cycles and explodes memory. vLLM’s continuous batching admits new requests every step. For DeepSeek V3 this matters because expert routing produces uneven per-token compute; some sequences finish early and free experts.

If you roll your own scheduler, you lose 30-50% throughput to idle expert shards. Use a mature engine.

Prefix caching flips the economics

Most RAG and agent workloads share a system prompt or retrieved context. With --enable-prefix-caching, the first request pays attention cost; later requests reuse the latent KV. In a batch of 64 with identical 2K prefix, you amortize that prefix to near zero. A DeepSeek V3 throughput benchmark that disables this feature understates real gains by 2-4x on shared-prefix traffic.

FP8 and precision: free throughput if your stack supports it

DeepSeek V3 was trained with FP8 mixed precision, and weights distribute cleanly to FP8 inference on H100/H800. Running the benchmark in BF16 duplicates memory traffic versus FP8 with negligible quality drop for most tasks. In a batch setting, FP8 reduces HBM reads for expert weights, directly lifting the tokens/sec ceiling. If your vLLM build supports --dtype fp8, include both runs in the DeepSeek V3 throughput benchmark. The delta is not a marketing trick; it is less data movement on the same silicon.

But watch out: some custom CUDA graphs and older drivers mishandle scaling factors. Validate accuracy on a held-out set before trusting the gain.

Expert parallelism across nodes

At 671B, even TP8 on one node leaves each GPU holding ~84B params (BF16 ~168GB, exceeding H100 80GB). Tensor parallelism alone forces pipeline or expert parallelism, or quantization. DeepSeek V3’s 256 experts (8 active per token) map naturally to expert-parallel sharding: assign expert subsets to different nodes. This changes the topology: cross-node all-to-all for router dispatch becomes the bottleneck, not matmul. A single-node FP8 run is the clean baseline; a multi-node EP run needs NIC bandwidth profiling.

If you only have one node, you must quantize to FP8 or use pipeline parallelism with microbatching, which hurts the batch throughput curve because pipeline bubbles grow with batch size.

Output length variance

Batched benchmarks often fix max_tokens equal across requests. Production has variance: some completions stop at 50 tokens, others hit 2K. Continuous batching handles this, but your aggregate throughput metric should weight by actual generated tokens, not requested cap. Otherwise you overestimate by the fraction of unused quota. In the script above, set max_tokens to a hard cap but measure usage.completion_tokens from each response.

Tradeoffs: latency, cost, and the batch knob

Maximizing throughput degrades time-to-first-token (TTFT). At batch 64, TTFT can climb from tens of milliseconds to multiple seconds because the scheduler queues requests behind longer sequences. For offline summarization, that is fine. For chat, users notice.

Cost tracks throughput inversely: higher batch efficiency means lower $/million tokens on self-hosted iron. But if you cap batch size for latency, you pay for idle GPUs. The decisive lever is separating synchronous interactive traffic from asynchronous bulk traffic into different endpoints with different --max-num-seqs.

Gateway overhead is real but small

If you consume DeepSeek V3 through a gateway like n4n.ai, which exposes an OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, add a modest network and routing overhead to local numbers. In practice that is 5-15% on throughput if you keep connections persistent and batch at the client. The benefit is that a provider rate limit or outage doesn’t zero your pipeline; the gateway reroutes. Honor its cache-control hints to preserve prefix benefits across hops.

Honest limitations of any public benchmark

Any DeepSeek V3 throughput benchmark published by a vendor runs on a curated mix. They will not show the 5% of requests that trigger expert imbalance and stall the all-to-all. They will not show tail latency at p99. Your job is to replicate the p50/p90/p99 split. Use a log replay from your own traffic if possible.

Decisive takeaway

Run your own DeepSeek V3 throughput benchmark with your real prompt distribution, enable prefix caching, and sweep concurrency until you see the knee. Expect MoE + MLA to reward batching further than dense models, but respect KV memory at long context. Split interactive and batch traffic into separate schedulers. If you run FP8, validate accuracy before shipping. If you use a hosted route, account for fallback overhead but lean on it for resilience. The model is engineered for high batch efficiency; your measurement discipline determines whether you capture it.

Tagsdeepseek-v3throughputbatch-inferencebenchmark

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceMeasuring throughput ceiling on 8x H100 clustersPractical analysis of measuring the real 8x H100 cluster throughput ceiling for batch inference: parallelism, saturation testing, and tradeoffs.Benchmarks & performanceThroughput benchmark: single-GPU vs multi-GPU inferencePractical comparison of single-GPU vs multi-GPU throughput for LLM inference: scaling efficiency, cost, latency, and which setup to pick for your workload.Benchmarks & performanceTokens per second at batch size 1 vs batch size 64A practical tokens per second batch size comparison: batch size 1 versus 64 across latency, cost, ergonomics, and limits for LLM inference.Benchmarks & performanceWhy larger batches don't always mean higher throughputExplains why increasing batch size eventually stops improving LLM inference throughput, with concrete tradeoffs between latency and hardware limits.Benchmarks & performanceBatch inference throughput benchmark on H100 vs A100Direct H100 vs A100 throughput benchmark for batch LLM inference across capabilities, cost, latency, and ergonomics, with a head-to-head table and verdict.Benchmarks & performanceBatch inference throughput for Llama 3.1 70B at scaleAnalysis of Llama 3.1 70B throughput at scale: continuous batching, KV cache limits, and parallel strategies that decide batch inference cost and latency.

All batch inference throughput benchmarks posts →