Quantization is the lever that makes large language models run on commodity hardware. The three formats you’ll encounter in production — GGUF, GPTQ, and AWQ — each make different trade-offs between compression ratio, inference latency, hardware support, and ecosystem maturity. Choosing the wrong one costs you either VRAM headroom or tokens per second. Here’s how to decide.
What each format actually does
GGUF (GPT-Generated Unified Format) is a single-file container that stores quantized weights alongside metadata, tokenizer config, and architecture hints. It was built for llama.cpp and CPU-first inference. Quantization happens post-training via k-means clustering on weight rows (the “k-quants” like Q4_K_M, Q5_K_S). The format supports mixed precision per-layer and runs on virtually anything with a CPU — Apple Silicon, x86, ARM, even WebAssembly.
GPTQ (Gradient-based Post-Training Quantization) uses one-shot weight reconstruction via approximate second-order optimization. It quantizes weights to INT4 or INT3 while keeping activations in FP16 at inference time. The original implementation targets CUDA kernels; you need a GPU to see the throughput gains. GPTQ models ship as Safetensors with a quantization_config block.
AWQ (Activation-aware Weight Quantization) observes that not all weights matter equally — only those multiplied by large-magnitude activations. It searches for a per-channel scaling factor that protects salient weights, then quantizes the rest to INT4. Like GPTQ, AWQ assumes GPU inference with FP16 activations. The calibration step requires a small representative dataset (typically 128–512 sequences).
Hardware support and runtime reality
| Dimension | GGUF | GPTQ | AWQ |
|---|---|---|---|
| Primary target | CPU, Apple Metal, Vulkan | NVIDIA CUDA | NVIDIA CUDA |
| GPU offload | Partial (llama.cpp n-gpu-layers) |
Native | Native |
| CPU inference | First-class | Slow (no optimized kernels) | Slow (no optimized kernels) |
| Apple Silicon | Metal backend, unified memory | Not supported | Not supported |
| AMD ROCm | Experimental | Via AutoGPTQ rocm branch | Via autoawq rocm branch |
| Web / edge | WASM, iOS, Android | No | No |
| Kernel maturity | High (llama.cpp, ctransformers) | High (AutoGPTQ, ExLlamaV2) | Medium (autoawq, vLLM) |
If you’re deploying on MacBooks, Raspberry Pis, or browser tabs, GGUF is the only serious option. The llama.cpp Metal backend saturates unified memory bandwidth on M-series chips — a 7B Q4_K_M model hits 30–45 tok/s on an M2 Max with 32 GB.
For NVIDIA GPUs, GPTQ and AWQ both run FP16 activations with INT4 weights, but the kernel implementations differ. ExLlamaV2 (GPTQ) uses custom CUDA kernels that fuse dequantization and GEMM, achieving near-FP16 throughput on H100/A100. AWQ in vLLM uses Marlin kernels — also fast, but the calibration step adds operational friction.
Model availability and conversion friction
Hugging Face hosts thousands of pre-quantized models for each format. Search gguf + model name and you’ll find TheBloke, MaziyarPanahi, or official org repos with Q4_K_M, Q5_K_M, Q8_0 variants. GPTQ and AWQ repos follow similar naming (-GPTQ-4bit-128g, -AWQ-4bit).
Converting yourself:
# GGUF via llama.cpp (CPU, no GPU needed)
python convert-hf-to-gguf.py /path/to/model --outfile model.q4_k_m.gguf --outtype q4_k_m
# GPTQ via AutoGPTQ (requires GPU VRAM ~2x model size)
python -m auto_gptq.cli.quantize \
--model /path/to/model \
--quant_path /path/to/output \
--bits 4 --group_size 128 --damp_percent 0.01
# AWQ via autoawq (requires calibration data)
python -m awq.quantize \
--model_path /path/to/model \
--quant_path /path/to/output \
--w_bit 4 --q_group_size 128 \
--calib_data wikitext2 --n_calib 512GGUF conversion runs on a laptop. GPTQ and AWQ need a GPU with enough VRAM to hold the FP16 model plus workspace — roughly 2× the model size. For a 70B model, that’s 140 GB+ VRAM (multi-GPU or H100). Most teams download pre-quantized artifacts instead.
Inference latency and throughput
On an H100 with vLLM, a 7B INT4 model serves roughly:
- GPTQ (ExLlamaV2 kernels): 2,800–3,200 tok/s single request, 12,000+ tok/s batched
- AWQ (Marlin kernels): 2,500–2,900 tok/s single request, 11,000+ tok/s batched
- GGUF (llama.cpp, GPU offload): 1,800–2,200 tok/s single request, 6,000–8,000 tok/s batched
The gap comes from kernel fusion. ExLlamaV2 and Marlin fuse dequant + GEMM + activation in one kernel launch. llama.cpp’s GPU offload still launches separate kernels per layer. On CPU, GGUF wins by default — the others don’t run.
Batch size matters. At batch=1, kernel launch overhead dominates. At batch=32+, all three saturate compute. If your workload is high-concurrency chat, the difference narrows. If it’s single-stream code generation, GPTQ/AWQ hold a 30–40% edge on NVIDIA.
Memory footprint
Quantization target is usually INT4 (4-bit weights). Actual disk and VRAM usage:
| Model (params) | FP16 baseline | GGUF Q4_K_M | GPTQ 4bit-128g | AWQ 4bit-128g |
|---|---|---|---|---|
| 7B | 14 GB | ~4.1 GB | ~3.8 GB | ~3.8 GB |
| 13B | 26 GB | ~7.6 GB | ~7.1 GB | ~7.1 GB |
| 34B | 68 GB | ~19.5 GB | ~18.2 GB | ~18.2 GB |
| 70B | 140 GB | ~39 GB | ~36.5 GB | ~36.5 GB |
GGUF overhead comes from metadata, tokenizer, and per-layer quantization maps. GPTQ/AWQ Safetensors are leaner — just quantized weights, scales, and zeros. For GPU deployment, the difference is ~0.3–0.5 GB on 7B, negligible on 70B. For CPU/edge, GGUF’s self-contained file simplifies distribution.
Quality at ISO quantization level
At 4-bit, all three formats reach ~98–99% of FP16 perplexity on standard benchmarks (WikiText2, C4, MMLU). The differences are smaller than variance across random seeds.
- GGUF k-quants (Q4_K_M, Q5_K_M) use mixed precision: attention weights at 4-bit, FFN at 5/6-bit, outliers at 8-bit. This preserves quality better than uniform INT4.
- GPTQ with
group_size=128anddamp_percent=0.01is the de facto standard. Smaller group_size (32, 64) improves quality but increases metadata overhead. - AWQ typically matches or slightly beats GPTQ at same bit-width because it protects salient channels. The gain is ~0.01–0.03 perplexity points — measurable, rarely user-visible.
At 3-bit, AWQ degrades less than GPTQ. At 8-bit, all are indistinguishable from FP16. If you need 3-bit for VRAM reasons, AWQ is the safer bet. If you need 4-bit, pick based on hardware.
Ecosystem and tooling maturity
GGUF: llama.cpp is the reference runtime. Bindings exist for Python (llama-cpp-python), Go, Rust, Node, Swift, C#. ctransformers provides a unified API. ollama and llamafile package GGUF for zero-config local inference. The format is stable — a GGUF from 2023 loads today.
GPTQ: AutoGPTQ for quantization, ExLlamaV2 / vLLM / TGI for serving. The Safetensors quantization_config schema is standardized. Model cards usually specify --group_size and --damp_percent used. Version drift is low.
AWQ: autoawq for quantization, vLLM / TGI for serving. Calibration data choice affects quality — wikitext2, c4, or domain-specific. Less standardized than GPTQ; some repos omit calibration details. Fewer quantization-time knobs.
All three work with OpenAI-compatible servers. n4n.ai routes to backends serving any of these formats without client changes — the gateway normalizes the /v1/chat/completions contract regardless of quantization underneath.
Operational considerations
Distribution: GGUF is one file. Copy it, run it. GPTQ/AWQ are Safetensors shards (usually 2–8 files) plus config.json and tokenizer.*. Container images for GPTQ/AWQ are larger.
Version pinning: llama.cpp breaks GGUF compatibility rarely (once per year). Pin the converter and runtime together. GPTQ/AWQ depend on transformers + auto_gptq / autoawq versions — pin all three.
Observability: GGUF exposes per-layer quantization via metadata. GPTQ/AWQ require parsing quantization_config. For per-token usage metering, all three work the same at the API layer.
Fallback behavior: If a GPU backend OOMs or degrades, CPU fallback only works transparently with GGUF. GPTQ/AWQ on CPU fall back to slow PyTorch kernels — expect 10–50× slowdown. Design your routing accordingly.
Which to choose
Local-first, CPU, Apple Silicon, edge, or heterogeneous fleet
GGUF. No GPU required. Runs on developer laptops, CI runners, mobile devices, browsers via WASM. Single-file distribution. Use Q4_K_M for 7B–13B, Q5_K_M for 34B+, Q8_0 if quality is paramount and RAM allows.
High-throughput NVIDIA GPU serving (vLLM, TGI, ExLlamaV2)
GPTQ for operational simplicity. Pre-quantized artifacts are abundant. ExLlamaV2 kernels are the fastest INT4 path on Hopper/Ampere. No calibration step. Standardized quantization_config makes CI validation trivial.
High-throughput NVIDIA GPU with quality sensitivity at 3-bit or aggressive compression
AWQ. The activation-aware scaling preserves perplexity better at 3-bit. At 4-bit, the edge is marginal — choose AWQ only if you already have a calibration pipeline and the 0.01 perplexity gain matters for your eval suite.
Multi-GPU 70B+ on limited VRAM (2× A100 80GB, 4× A10G 24GB)
GPTQ or AWQ with group_size=128. Both shard cleanly across GPUs via tensor parallelism in vLLM/TGI. GGUF multi-GPU support in llama.cpp is experimental and slower.
One model artifact across cloud GPU and local CPU
GGUF. Serve the same .gguf file from a GPU box (with n-gpu-layers=99) and a CPU-only fallback. No dual-publish pipeline.
Quantizing a custom fine-tune yourself, no calibration data handy
GPTQ. One-shot, no calibration set required. Run auto_gptq on a single GPU overnight. AWQ needs representative data — if your domain differs from WikiText2, you must curate it.
Final verdict
Default to GGUF for development, edge, and any CPU target. Default to GPTQ for production GPU serving on NVIDIA. Reach for AWQ only when pushing 3-bit quantization or when your eval suite proves a measurable quality delta at 4-bit that justifies the calibration pipeline.
The format matters less than the quantization parameters. A well-tuned GPTQ 4-bit-128g beats a poorly calibrated AWQ 4-bit. A GGUF Q5_K_M beats a GGUF Q4_0. Start with the format your hardware demands, then tune the knobs.