The qwen2.5 model size speed curve is the single most important constraint when you pick a model for production inference. Across the 0.5B to 72B range, token throughput falls off faster than parameter count grows, while quality gains flatten after the mid-size tiers.
The hardware reality behind the curve
Parameter count is a proxy for memory footprint, not compute alone. At fp16, every billion parameters costs 2GB. A 0.5B model sits under 1.5GB; a 72B model needs ~144GB just for weights. That difference dictates the hardware tier: the small end runs on a Raspberry Pi or laptop CPU, the top end demands multi-GPU nodes with NVLink or fast PCIe.
Decode-phase inference is memory-bandwidth bound. Each generated token requires a full weight read from VRAM to compute units. So speed scales inversely with model size on a fixed memory bus. A 7B model on an A100 (2TB/s bandwidth) can sustain hundreds of tokens/s for a single stream; a 72B model on the same card reads 10x the data per token, dropping to tens of tokens/s before accounting for tensor parallelism overhead.
Tensor parallelism splits weights across GPUs, but introduces all-reduce communication. At 72B across 4 GPUs, the per-token sync eats 10–20% of the wall-clock, worsening the already steep qwen2.5 model size speed curve.
Token throughput vs model size
The curve is steepest between 7B and 32B. Below 7B, models are small enough that compute units stall waiting for kernels, not weights. Above 32B, you pay linear memory cost but get diminishing quality returns for many tasks.
Public MLPerf-style runs and community llama.cpp reports show a consistent pattern: on identical consumer hardware (e.g., RTX 4090, 24GB), a 7B Q4_K_M quant yields ~40–60 tokens/s, a 14B around 20–30, a 32B barely fits at Q4 and drops to <10. The 72B is off the table without split layers across two cards. On datacenter A100/H100, the absolute numbers rise but the ratios hold: a 72B is roughly an order of magnitude slower per stream than a 7B.
Quantization changes the equation
Weight-only quantization (INT8, INT4) compresses the memory curve. A 72B at Q4 fits in ~36GB, making it viable on a single 48GB A6000 or dual 24GB cards. But quantization does not shrink the compute for attention or the activation memory; it mainly reduces weight fetch bandwidth.
Thus the qwen2.5 model size speed curve flattens somewhat with INT4: a 72B-Q4 can approach the token rate of a 32B-fp16 on the same bus. However, you trade minor accuracy and risk outlier degradation on reasoning tasks. For 0.5B and 1.5B, quantization is almost free—they already fit everywhere and CPU inference stays snappy.
# llama.cpp example, real flags
./llama-cli -m qwen2.5-7b-q4_k_m.gguf -p "Summarize: ..." -n 256 -t 8Context length multiplies the penalty
Long prompts inflate the KV cache linearly with layers and hidden size. A 72B with 32k context can consume tens of GB just for KV at modest batch. That forces smaller batches, pushing the speed curve further down.
# per token KV bytes = 2 * n_layers * n_heads * head_dim * dtype_bytes
# Qwen2.5-72B: layers=80, heads=64, dim=8192 -> head_dim=128
kv_per_token = 2 * 80 * 64 * 128 * 2 # fp16
print(kv_per_token / 1024, "KB/token") # ~2560 KB/tokenAt 4k context and batch 32, that is 2.5MB * 4096 * 32 ≈ 327GB—clearly impossible, which is why paging (PagedAttention) and strict batch limits are mandatory at the top end. Smaller models have proportionally smaller KV: a 7B (layers=28, heads=28, dim=3584) uses ~0.3MB/token, an order of magnitude less pressure.
Latency under load: batching and concurrency
Single-stream latency hides the real cost. Production serves many concurrent requests. Larger models have larger KV caches; a 72B with 4k context uses per-token KV memory that multiplies fast. Continuous batching helps, but the maximum batch size before latency blows up is far smaller for 72B than 7B.
If you need 100 req/s at p95 < 200ms, a 7B on a single GPU beats a 72B on four GPUs on price and tail latency. The qwen2.5 model size speed curve for throughput per dollar is brutal at the top.
Quality per watt: where the curve bends
Qwen2.5 published scores show MMLU climbing from under 40% at 0.5B to ~60% at 7B, ~75% at 14B, and above 80% at 72B. Coding and reasoning follow similar shapes. But for extraction, classification, or templated generation, 3B often matches 72B within noise. The bend is around 7B–14B: beyond that you pay 5–10x infra for <10% quality lift on everyday tasks.
A practical benchmarking harness
Measure don’t assume. Use a consistent prompt and max_tokens, hit an OpenAI-compatible endpoint, and record tokens/s and TTFT (time to first token).
from openai import OpenAI
import time, statistics
client = OpenAI(base_url="https://gateway.example.com/v1", api_key="key")
def trial(model, prompt, n=5):
rates, ttfts = [], []
for _ in range(n):
t0 = time.perf_counter()
stream = client.chat.completions.create(
model=model, messages=[{"role":"user","content":prompt}],
max_tokens=128, stream=True)
first = None
tokens = 0
for chunk in stream:
if chunk.choices[0].delta.content:
if first is None:
first = time.perf_counter()
tokens += 1
ttfts.append(first - t0)
rates.append(tokens / (time.perf_counter() - first))
return statistics.mean(rates), statistics.mean(ttfts)
for m in ["qwen2.5-0.5b","qwen2.5-3b","qwen2.5-7b","qwen2.5-14b","qwen2.5-32b","qwen2.5-72b"]:
r, t = trial(m, "Write a SQL query for latest orders")
print(f"{m}: {r:.1f} tok/s, TTFT {t*1000:.0f}ms")When testing across providers, an OpenAI-compatible gateway such as n4n.ai that honors client routing directives and provides automatic fallback lets you run the same loop against different backends without rewriting the client. You can pin a fallback chain via a routing body field:
{
"model": "qwen2.5-72b",
"route": { "fallback": ["qwen2.5-32b"] },
"cache_control": { "ttl": 600 }
}This forwards provider cache-control hints and degrades gracefully if the 72B endpoint is saturated.
Tradeoffs by deployment scenario
- Edge / on-device: 0.5B–1.5B. CPU inference at 10–20 tokens/s is fine for autocomplete or intent detection. No GPU needed.
- Latency-sensitive API: 3B–7B. Single consumer GPU, sub-100ms TTFT, high concurrency.
- Balanced backend: 14B–32B. Datacenter GPU, best quality/cost for agentic workflows.
- Async heavy reasoning: 72B. Batch overnight, long context, tolerate seconds of latency.
Decisive takeaway
Pick the smallest Qwen2.5 that meets your accuracy bar, then quantize aggressively. The qwen2.5 model size speed curve proves that 7B–14B covers 80% of production needs; reserve 32B+ for tasks where a few percent points of quality offset 10x infra spend. Benchmark on your own hardware with the harness above—don’t trust vendor charts.