Quantization impact on inference speed: DeepSeek V3

DeepSeek V3 quantization inference speed determines whether you can serve the 671B-parameter mixture-of-experts model at acceptable latency or drown in VRAM costs. The thesis here is simple: quantization is not a uniform win, and the right precision tier depends on your accelerator, batch size, and tolerance for quality regression.

Why DeepSeek V3 is sensitive to quantization

DeepSeek V3 uses a MoE layout with 671B total parameters but only 37B active per token. That ratio is the crux. All expert weights must reside in accelerator memory to avoid fetch-on-demand stalls, even though a single forward pass touches a fraction. Decode—the token-by-token generation phase—is almost entirely memory-bandwidth bound: you stream weights from HBM to compute units for every token produced.

Prefill (processing the prompt) is more compute bound because sequence length multiplies matmul work. Quantization helps both, but the mechanism differs. During decode, cutting weight precision from FP16 to FP8 halves the bytes moved per parameter. During prefill, lower precision also enables higher throughput if the tensor cores can consume the narrow types natively.

The model was trained with FP8 master weights, so inference at FP8 is the natural operating point. Dropping to INT8 or INT4 is a post-training choice that trades precision for footprint.

Quantization schemes that apply to V3

FP8 (E4M3/E5M2)

This is the baseline you should measure against. Hopper (H100) and Blackwell have native FP8 tensor cores. Weight-only FP8 or FP8 activation keeps the math close to training. Memory per weight drops from 2 bytes (FP16) to 1 byte.

INT8 weight-only

Common on Ampere and earlier. Weights are stored as int8 and dequantized to FP16 before matmul if the kernel doesn’t support INT8 compute. You still get the memory reduction but compute may not accelerate.

INT4 (GPTQ/AWQ)

Stores weights at 4 bits. Usually requires dequant to FP16/FP8 for the actual multiply unless you have INT4 tensor cores (Ada Lovelace, some edge accelerators). Memory drops to 0.5 bytes/weight versus FP16.

Expert-wise mixed precision

Because MoE experts are independent, you can quantize rare experts more aggressively than heavily used ones. This is an emerging practice, not yet a stable off-the-shelf flag in most servers.

Expected inference speed impact

Start with the decode bandwidth bound. For a single stream on an H100 (3.35 TB/s HBM3), moving FP16 weights for 37B active params costs:

37B params * 2 bytes = 74 GB per token
3350 GB/s / 74 GB ≈ 45 tokens/s (ideal, no overhead)

Drop to FP8 and the same math gives ~90 tokens/s ideal. INT4 lands near ~180 tokens/s theoretical, but only if the dequant overhead is hidden or the hardware multiplies in INT4. Real numbers are lower due to kernel launch, routing, and KV cache traffic, but the trend holds: DeepSeek V3 quantization inference speed scales near-linearly with weight bytes reduced during decode.

At large batch sizes, utilization shifts. With continuous batching, the GPU saturates compute during prefill and matmul-heavy decode. Here, FP8 wins on both bandwidth and compute throughput. INT8 without INT8 compute sees smaller gains because the matmul executes in FP16 after dequant. INT4 may even regress per-token latency if dequant is on the critical path.

A typical vLLM launch for an FP8 checkpoint looks like:

vllm serve deepseek-ai/DeepSeek-V3 \
  --quantization fp8 \
  --tensor-parallel-size 8 \
  --max-model-len 8192

For INT4 you would swap the flag and ensure the checkpoint is already GPTQ-converted.

Quality tradeoffs

FP8 is within noise on perplexity and downstream coding/math benchmarks for V3—the training pipeline already used it. INT8 weight-only typically loses <1% on MMLU-style tasks but can nibble at long-context reasoning. INT4 is where you must evaluate. We have seen 2–4 point drops on complex multi-step math and occasional formatting drift in code generation.

The decisive test is task-specific. If you serve chat summarization, INT4 may be invisible. If you serve agentic tool use with DeepSeek V3, INT4 can break parameter extraction. Always run a held-out eval set before promoting a quantized build.

Serving considerations

KV cache precision is a separate lever. DeepSeek V3’s long context means KV cache can dominate memory at high concurrency. Using FP8 or INT4 KV cache (supported in some servers) compounds the footprint win but adds another quality variable.

Routing matters when you host multiple precisions. An OpenAI-compatible gateway such as n4n.ai can honor a client routing directive that pins a request to a specific quantized variant by model name, letting you shift traffic between FP8 and INT4 builds without code changes:

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"deepseek-v3-int8","messages":[{"role":"user","content":"Explain MoE routing"}]}'

This is useful for canary-testing INT4 on a fraction of traffic while the majority hits FP8.

Another concrete point: tensor parallelism interacts with quantization. At TP=8 on A100s, INT8 weight-only fits V3 in 640GB aggregate (80GB*8) versus FP16’s 1.34TB requirement that needs TP=16 or host offload. The speed win is partly from simply being able to run without NVLink-bound CPU fallback.

Honest limitations

Quantization does not fix attention bottlenecks. If your prompt is 32K tokens, prefill time is governed by sequence length and head dimension regardless of weight precision. Similarly, small batch interactive traffic gets the full bandwidth benefit; large batch throughput is capped by compute units that may not natively execute sub-8-bit.

Also, not all inference servers support all schemes for MoE cleanly. Expert shuffling across TP ranks can break naive INT4 kernels. Validate with a real load test, not just a single-token benchmark.

Takeaway

For DeepSeek V3 quantization inference speed, the priority order is clear:

1. Run FP8 if you have Hopper/Blackwell. It is the native dtype, halves memory, doubles decode bandwidth headroom, and preserves quality.
2. Use INT8 weight-only on Ampere when FP8 isn’t available. You get the memory win; expect modest compute speedup unless INT8 kernels are active.
3. Reserve INT4 for high-volume, low-complexity paths and only after a task eval confirms acceptable quality. Treat it as a cost lever, not a default.

DeepSeek V3 is large enough that quantization is mandatory for economical serving, but the precision tier is an engineering decision, not a checkbox. Measure decode tokens/s at your target batch, measure quality on real prompts, and route accordingly.