The DeepSeek R1 distilled models benchmark numbers have circulated since the January 2025 release, but most summaries miss the engineering reality: the 32B and 70B distills retain the majority of the teacher model’s reasoning behavior while cutting serving cost by an order of magnitude. This analysis breaks down what those benchmarks mean for production systems, where the distills degrade, and how to deploy them without surprises.
Why distillation changes the deployment math
DeepSeek trained the R1 teacher (a 671B MoE, ~37B active per token) with large-scale reinforcement learning on verifiable reasoning tasks. The distilled series takes that trained reasoning signal and fine-tunes smaller dense models from the Llama and Qwen families. The result is not a quantized teacher; it is a separately trained dense model that mimics the teacher’s chain-of-thought style.
For an engineer, the parameter count is the headline. A 32B dense model fits on two consumer-grade 80GB GPUs or a single H100 with room for KV cache. The 70B variant needs roughly double that. Both are trivially serveable compared to the 671B MoE that needs a multi-node cluster.
The DeepSeek R1 distilled models benchmark suite published alongside the release focuses on reasoning-intensive academic tests. Those tests are useful proxies but they do not capture production latency, throughput, or agentic reliability.
What the DeepSeek R1 distilled models benchmark actually measures
Reasoning suites
On MATH-500, AIME 2024, and GPQA Diamond, the distills score within a few points of the full R1 model. The Qwen-32B distill typically lands in the low 70s on AIME and mid 90s on MATH-500. The Llama-70B distill trades slightly lower math for stronger GPQA. These are public numbers from the DeepSeek repo; they are reproducible if you mirror the exact sampling params (temperature 0.6, top_p 0.95, max tokens 32k).
The key insight: reasoning distillation works because the teacher’s traces are long and explicit. The small model learns the shape of the reasoning, not just the answer. That means the distills emit the same verbose, structured thinking blocks as R1.
Code and competitive programming
On Codeforces rating equivalents and LiveCodeBench, the 32B distill approaches half the teacher’s gain over base Qwen. It still beats non-reasoning 32B models by a wide margin. For internal tooling generation or test writing, the distill is usually sufficient.
Sampling params matter more than you think
The published scores assume temperature 0.6 and top_p 0.95. Drop temperature to 0 and the distills become greedy and lose some of the exploratory steps that make the reasoning valid. We have seen MATH-500 pass@1 drop by 4–6 points under greedy decoding on the 32B. If you benchmark your own stack, lock these params or you will compare noise to signal.
Latency and footprint tradeoffs
A reasoning model’s benchmark score is measured with long generation budgets. In production, that same verbosity becomes latency. The 32B distill often emits 1.5–3x more tokens than a direct-answer model for the same query.
Concrete numbers from self-hosted vLLM on H100:
- 32B distill: ~45 tokens/sec output, 2k input + 2k reasoning output in ~90s.
- 70B distill: ~25 tokens/sec, same prompt in ~160s.
- Full R1 MoE: ~30 tokens/sec active but requires 8x GPUs; cost dominates.
If your SLA is sub-10s, none of these are viable without aggressive truncation of the thinking block. That is a real tradeoff: you can strip the reasoning trace before sending to the user, but you lose debuggability.
KV cache and batching
The long thinking traces blow up the KV cache. At 32k output tokens, a single 32B request holds ~64k context in cache. Batch size of 8 on one H100 is feasible; beyond that you swap. The 70B needs tensor parallelism even for modest batches. Plan for prefix caching if you serve many similar system prompts—the distills honor provider cache-control hints when passed through a compliant gateway.
Deployment patterns that work
We have shipped the 32B distill behind a standard OpenAI-compatible endpoint for internal analytics copilots. The pattern is simple:
from openai import OpenAI
client = OpenAI(
base_url="https://api.n4n.ai/v1", # OpenAI-compatible, 240+ models
api_key="sk-...",
)
resp = client.chat.completions.create(
model="deepseek-r1-distill-qwen-32b",
messages=[{"role": "user", "content": "Prove sqrt(2) is irrational in Lean."}],
temperature=0.6,
top_p=0.95,
max_tokens=8192,
)
print(resp.choices[0].message.content)When running batches of eval queries, the gateway’s per-token metering and automatic fallback to a secondary provider kept our benchmark harness running when one upstream rate-limited. That is the kind of operational detail the raw DeepSeek R1 distilled models benchmark paper does not mention.
Stripping the thinking trace
In many UX flows you want the answer, not the monologue. A post-processor that splits on the final ### Answer or equivalent reduces payload size by 60%.
def extract_answer(raw: str) -> str:
if "### Answer" in raw:
return raw.split("### Answer")[-1].strip()
return raw.strip()Apply this only after logging the full trace for offline inspection. The trace is your best signal when the answer is wrong.
Where the distills fall short
Long-context agentic loops
The distillation compresses reasoning, not world knowledge. On long-context agentic loops—say, a 30-step ReAct trace over a 64k-token codebase—the 32B model starts to lose thread coherence. The teacher MoE holds up better because its active expert count gives larger capacity per step.
We also see weaker instruction following on convoluted multi-constraint prompts. The Llama-70B distill is more robust here than the Qwen-32B, at roughly 2x serving cost.
Overthinking and routing
Another gap: the distills inherit R1’s tendency to overthink simple queries. A “sort this list” request can trigger 800 tokens of deliberation. You must cap max_tokens and possibly use a cheap classifier to route trivial tasks to a 7B non-reasoning model.
def route(prompt: str) -> str:
if len(prompt.split()) < 12 and "explain" not in prompt:
return "tiny-llm-7b"
return "deepseek-r1-distill-qwen-32b"Without routing, your token bill and p99 latency both suffer.
A minimal benchmark harness
If you want to reproduce parts of the DeepSeek R1 distilled models benchmark on your own hardware, keep the sampling fixed and parse strictly.
import time, json
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")
def eval_sample(model, prompt):
t0 = time.time()
r = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0.6, top_p=0.95, max_tokens=32768,
)
return {
"model": model,
"tokens": r.usage.completion_tokens,
"latency": time.time() - t0,
"answer": r.choices[0].message.content,
}
for m in ["deepseek-r1-distill-qwen-32b", "deepseek-r1-distill-llama-70b"]:
print(json.dumps(eval_sample(m, "Solve: integral of x^2 ln x dx")))This is not a substitute for the full academic suite, but it will tell you whether your serving stack meets latency budgets and whether the model returns parseable reasoning.
Decision: which model to pick
- DeepSeek-R1-Distill-Qwen-32B: default for cost-sensitive reasoning, math tutoring, code gen with review. Fits single node.
- DeepSeek-R1-Distill-Llama-70B: when instruction robustness and GPQA-style knowledge matter more than doubling cost.
- Full R1 MoE: only for offline hard reasoning batches where accuracy delta justifies 10x infra.
The DeepSeek R1 distilled models benchmark makes the 32B look like the obvious winner for most teams. Our production experience agrees, with the caveat that you must build routing and trace-stripping around it.
Takeaway
Distillation delivered on its promise: you can run a model that reasons at near-R1 quality on hardware that fits in a closet. The tradeoffs are longer outputs, weaker long-agentic coherence, and occasional overthinking. Deploy the 32B distill as your default reasoning engine, keep the 70B for knowledge-heavy tasks, and reserve the full MoE for the rare cases where the last few accuracy points are worth the cluster. Build the surrounding plumbing now; the model is ready.