When you’re picking a model file for local inference, the filename suffix does a lot of heavy lifting. Understanding GGUF quantization levels explained in practical terms — not just bit counts — saves you from downloading a 40 GB model that your 24 GB VRAM can’t run, or a 4-bit model that hallucinates on your coding tasks. The three most common choices you’ll see on Hugging Face are Q4_K_M, Q5_K_M, and Q8_0. They represent fundamentally different trade-offs, and the right one depends on your hardware, your quality bar, and whether you’re optimizing for throughput or latency.
What GGUF quantization actually does
GGUF (GPT-Generated Unified Format) is the container format used by llama.cpp and its derivatives. Quantization in this context means taking the original fp16 or bf16 weights and compressing them to lower precision with a calibration step that minimizes accuracy loss. The “K” in Q4_K_M and Q5_K_M stands for k-means: the quantization buckets weight values into clusters rather than using uniform linear spacing. This preserves outliers — the few weights that matter disproportionately for model quality — better than the older Q4_0/Q5_0 schemes.
Q8_0 is simpler: it quantizes every tensor to 8-bit integers with a per-block scale factor. No k-means, no mixed precision. It’s effectively a drop-in replacement for fp16 at half the memory cost, with negligible quality loss on most benchmarks.
The “_M” suffix means “medium” — it uses a mixed quantization strategy where attention tensors get slightly higher precision (Q6_K for Q4_K_M, Q6_K for Q5_K_M) while feed-forward layers stay at the base quantization. This targets the layers most sensitive to quantization error.
The three levels compared
| Dimension | Q4_K_M | Q5_K_M | Q8_0 |
|---|---|---|---|
| Effective bits/weight | ~4.25 (mixed) | ~5.25 (mixed) | 8.0 (uniform) |
| Model size (7B params) | ~3.8 GB | ~4.7 GB | ~7.2 GB |
| Model size (70B params) | ~38 GB | ~47 GB | ~72 GB |
| VRAM needed (7B, 4k ctx) | ~5.5 GB | ~6.5 GB | ~9.5 GB |
| VRAM needed (70B, 4k ctx) | ~42 GB | ~51 GB | ~78 GB |
| Perplexity delta vs fp16 | +0.15–0.30 | +0.05–0.15 | +0.01–0.03 |
| Token throughput (relative) | 1.0x (baseline) | ~0.85x | ~0.65x |
| First-token latency (relative) | 1.0x | ~1.1x | ~1.3x |
| Quality cliff risk | Moderate (reasoning, code) | Low | Negligible |
| Hardware sweet spot | 8–16 GB VRAM | 16–24 GB VRAM | 24–48 GB VRAM |
Memory and disk footprint
The size differences are not academic. A 7B model at Q4_K_M fits comfortably on an 8 GB GPU with room for a 4k–8k context window. The same model at Q8_0 needs ~9.5 GB just for weights, pushing you to 12 GB+ for any real workload. For 70B models, Q4_K_M is the only one that fits on a single 48 GB card (or dual 24 GB with tensor splitting); Q5_K_M requires 80 GB aggregate VRAM; Q8_0 needs 80 GB+ and is practically multi-GPU only.
Disk I/O matters too. llama.cpp mmaps the model file. On NVMe, the difference between 3.8 GB and 7.2 GB is a few hundred milliseconds at startup. On a slow USB drive or network mount, Q8_0 adds seconds to cold start. If you’re running ephemeral containers or serverless functions, Q4_K_M’s smaller footprint reduces cold-start variance.
Inference speed and throughput
Counterintuitively, lower quantization often means higher throughput on memory-bound hardware. The bottleneck for local LLM inference is almost always memory bandwidth, not compute. Q4_K_M moves ~45% less data per token than Q8_0. On an RTX 3090 (936 GB/s), that translates to roughly 1.4–1.5x higher tokens/second for Q4_K_M vs Q8_0 at batch size 1.
But there’s a catch: the k-means dequantization kernel for K-quants is more complex than Q8_0’s simple multiply-add. On Apple Silicon (unified memory, high bandwidth, AMX accelerators), the throughput gap narrows. On M2 Max, Q4_K_M is only ~1.15x faster than Q8_0. On CPU-only inference (AVX2/AVX-512), Q8_0 can actually edge out Q4_K_M because the dequantization overhead dominates when memory bandwidth isn’t the limiter.
First-token latency (prefill) scales differently. It’s compute-bound for the prompt processing phase. Q8_0’s simpler kernels give it an advantage here at large batch sizes, but for typical single-user interactive workloads (batch=1), the memory bandwidth win still favors Q4_K_M.
Quality degradation patterns
The perplexity numbers in the table are averages. The distribution of errors matters more for real use cases.
Q4_K_M shows measurable degradation on:
- Multi-step reasoning (GSM8K, MATH)
- Code generation (HumanEval, MBPP) — especially syntax correctness in longer contexts
- Instruction following with complex constraints
- Low-resource languages
Q5_K_M closes most of this gap. In blind A/B tests, most engineers can’t distinguish Q5_K_M from fp16 on general chat and summarization. Code and reasoning still show a slight deficit, but it’s rarely the bottleneck.
Q8_0 is indistinguishable from fp16 on every standard benchmark. The only time you’ll see a difference is in very long-context tasks (32k+) where accumulated quantization error in attention scores can theoretically drift, but even that is speculative — no public eval shows a statistically significant gap.
If you’re building a coding assistant, Q4_K_M is risky. If you’re building a chatbot or RAG system with 4k context, Q4_K_M is usually fine. Q5_K_M is the safe default for “I don’t want to think about quality.”
Hardware considerations
NVIDIA consumer (RTX 30/40 series)
VRAM is the hard constraint. Q4_K_M lets you run 7B–13B on 8–12 GB, 34B on 24 GB. Q5_K_M pushes 13B to 16 GB, 34B to 32 GB (dual 3090/4090). Q8_0 restricts you to 7B on 12 GB, 13B on 24 GB. Tensor splitting across mismatched GPUs works but adds latency variance — avoid if possible.
NVIDIA datacenter (A100, H100)
You have bandwidth to burn. Q8_0 becomes attractive because you’re not memory-bound at small batch sizes. The simpler kernels utilize tensor cores more cleanly. At batch > 8, Q8_0 can match or exceed Q4_K_M throughput on H100.
Apple Silicon (M1/M2/M3 Max/Ultra)
Unified memory changes the calculus. No VRAM vs RAM distinction — you’re limited by total system RAM. Q4_K_M 70B runs on 64 GB MacBook Pro. Q5_K_M 70B needs 96 GB (M2 Ultra). Q8_0 70B needs 128 GB+. The AMX units accelerate INT8 well, so Q8_0 throughput is competitive. If you have 96 GB+, Q5_K_M is the sweet spot; 64 GB forces Q4_K_M.
CPU-only (x86_64, ARM)
AVX2/AVX-512 and NEON dot-product instructions favor Q8_0’s uniform quantization. The k-means lookup tables for K-quants don’t vectorize as cleanly. On a modern Xeon or EPYC, Q8_0 7B runs at ~15–20 tok/s; Q4_K_M runs at ~12–15 tok/s. The quality per watt favors Q8_0 here.
Ecosystem and tooling
All three levels are first-class citizens in llama.cpp, ollama, LM Studio, text-generation-webui, and ctransformers. No compatibility gaps.
The only practical difference: some older quantization scripts (pre-2024) don’t support K-quants natively. If you’re quantizing from scratch with llama-quantize, you need a recent build. Downloading pre-quantized GGUFs from TheBloke, bartowski, or MaziyarPanahi avoids this entirely — they all ship K-quants now.
One operational note: Q4_K_M and Q5_K_M use per-tensor quantization maps stored in the GGUF metadata. If you’re writing a custom loader or doing model surgery (merging LoRAs, splicing layers), you must preserve the quantization_version and block_count fields. Q8_0 is simpler — just scale + zero-point per block.
Which to choose
You have 8–12 GB VRAM / 16–32 GB unified memory
Choose Q4_K_M. It’s the only way to run 7B–13B models with usable context. Accept the quality hit on coding/reasoning, or use a stronger base model (e.g., Qwen2.5-7B-Instruct at Q4_K_M beats Llama-3-8B at Q4_K_M).
You have 16–24 GB VRAM / 48–64 GB unified memory
Choose Q5_K_M. You can run 13B–34B models at near-fp16 quality. This is the engineering sweet spot — best quality-per-GB for almost every workload. The 15% throughput penalty vs Q4_K_M is worth it.
You have 24–48 GB VRAM / 96+ GB unified memory / datacenter GPUs
Choose Q8_0 if you’re serving multiple concurrent users (batch > 4) or running long-context (16k+). The simpler kernels scale better under load, and quality is indistinguishable from fp16. For single-user interactive, Q5_K_M still wins on latency and leaves headroom for larger context.
CPU-only inference
Choose Q8_0. The throughput advantage is real, and you’re not VRAM-constrained. Disk space is cheaper than CPU cycles.
Building a coding assistant or reasoning-heavy agent
Minimum Q5_K_M. Q4_K_M’s syntax error rate on non-trivial functions is high enough to break user trust. If hardware forces Q4_K_M, pair it with a syntax-aware verifier (tree-sitter + type checker in the loop) — don’t serve raw output.
RAG / summarization / classification / chat
Q4_K_M is fine. These tasks are robust to quantization noise. The context window matters more than the last 0.1 perplexity points. Spend your VRAM budget on longer context, not higher precision.
Uncertain or mixed workloads
Default to Q5_K_M. It’s the lowest-regret choice. You won’t hit a quality cliff, and it runs on hardware that’s widely available (24 GB consumer, 48 GB pro, 64 GB Mac). If you later need more throughput, you can quantize down to Q4_K_M from the same fp16 source — no re-download needed.