How to choose a quantization format for local inference

Choosing a quantization format isn’t about chasing the smallest file size — it’s about matching the format to your hardware, your runtime, and the quality floor your application can tolerate. The wrong choice means silent degradation, crashes on load, or leaving 30% of your GPU idle. Here’s a repeatable process to get it right the first time.

Step 1: Identify your runtime and hardware constraints

Before comparing formats, lock down where the model runs. The runtime dictates which formats are even loadable.

If you’re on a MacBook Pro M-series, GGUF via llama.cpp or MLX is your only zero-friction path. If you have an RTX 3090/4090 and want maximum throughput, GPTQ or AWQ via ExLlamaV2 or vLLM wins. If you’re on AMD ROCm or a mixed fleet, GGUF is the only format with first-class support everywhere.

Action: Write down your runtime, GPU vendor, and VRAM budget. That alone eliminates half the formats.

Step 2: Decide your quality floor with a calibration set

Quantization error isn’t uniform. Some models tolerate 4-bit well; others collapse on coding or reasoning tasks. You need a small, representative eval set — 50-100 prompts covering your actual use cases — before you commit.

# eval_calibration.py
import json
from pathlib import Path
from llama_cpp import Llama

CALIBRATION_PROMPTS = [
    "Write a Python function that parses RFC 3339 timestamps.",
    "Explain the difference between eventual and strong consistency.",
    "Refactor this code to use async/await: ...",
    # ... 47 more from your real workload
]

def evaluate_model(model_path: str, n_gpu_layers: int = -1) -> dict:
    llm = Llama(model_path=model_path, n_gpu_layers=n_gpu_layers, n_ctx=4096, verbose=False)
    results = []
    for prompt in CALIBRATION_PROMPTS:
        out = llm(prompt, max_tokens=256, temperature=0.1, stop=["</s>"])
        results.append({"prompt": prompt, "response": out["choices"][0]["text"]})
    return results

if __name__ == "__main__":
    import sys
    results = evaluate_model(sys.argv[1])
    Path("eval_results.json").write_text(json.dumps(results, indent=2))

Run this against the full-precision model (or a high-bit baseline like Q8_0) to establish your quality ceiling. Then test each candidate quant. If Q4_K_M drops your pass rate from 92% to 78% on coding tasks but Q5_K_S holds at 89%, you’ve found your floor.

Verification: Save eval outputs per quant. Diff them. If you can’t automate quality checks, at minimum eyeball 20 samples per format.

Step 3: Match quantization scheme to model architecture

Not all quants are created equal for every architecture. The quantization method (GPTQ, AWQ, GGUF’s k-quants) interacts with the model’s weight distribution.

• LLaMA / Mistral / Mixtral / Qwen / Gemma: All three major formats work well. GGUF k-quants (Q4_K_M, Q5_K_M, Q6_K) are battle-tested. GPTQ and AWQ shine on GPU with ExLlamaV2/vLLM.
• Falcon / MPT / older GPT-NeoX: AWQ often fails calibration (activation outliers). GPTQ with group_size=128 and damp_percent=0.01 is safer. GGUF works but may need Q6_K for parity.
• MoE models (Mixtral, DeepSeekMoE): Expert routing makes activation-aware quantization (AWQ) theoretically attractive, but in practice GGUF Q4_K_M or GPTQ 4-bit with group_size=64 on the experts works fine. AWQ calibration time scales poorly with expert count.
• Vision-language models (LLaVA, Qwen-VL): The vision tower usually stays FP16. Quantize only the LLM backbone. GGUF supports this natively via --mmproj split. GPTQ/AWQ tooling often quantizes the whole thing unless you surgically exclude modules.

# GGUF: quantize only the language model, keep vision tower FP16
llama-quantize llava-v1.5-7b.f16.gguf llava-v1.5-7b-q4_k_m.gguf Q4_K_M
# The mmproj file stays separate and unquantized

Rule of thumb: Start with GGUF Q4_K_M. It’s the most portable baseline. Only move to GPTQ/AWQ if you need GPU throughput that llama.cpp can’t deliver.

Step 4: Choose the specific quantization level

Within a format, the bit-width and quantization strategy determine the quality/size/speed tradeoff.

GGUF k-quants (llama.cpp naming)

The _K variants use k-means clustering per-block; _M adds mixed-precision (important weights at higher bits). Avoid legacy q4_0, q5_0 — they’re strictly worse than k-quants at same size.

GPTQ / AWQ (AutoGPTQ / AutoAWQ naming)

Smaller group size = finer granularity = better quality = larger model = slower kernel. For 7B-13B models, 4bit-128g is the default. For 30B+, drop to 4bit-64g or 4bit-32g if quality suffers. AWQ only supports 4-bit and 3-bit; 4-bit-128g is the standard.

Action: For GGUF, start at Q4_K_M. For GPTQ/AWQ, start at 4bit-128g. Move up one notch only if eval fails.

Step 5: Generate or download the quantized artifact

Option A: Quantize from FP16/BF16 yourself (recommended)

You control the calibration data and verify the output. Never trust a random Hugging Face quant without checking its provenance.

GGUF via llama.cpp:

# 1. Convert HF -> GGUF FP16
python llama.cpp/convert_hf_to_gguf.py /path/to/model --outfile model.f16.gguf --outtype f16

# 2. Quantize to Q4_K_M
llama.cpp/quantize model.f16.gguf model.q4_k_m.gguf Q4_K_M

GPTQ via AutoGPTQ (requires CUDA):

# quantize_gptq.py
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

model_id = "mistralai/Mistral-7B-v0.1"
quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    damp_percent=0.01,
    desc_act=False,  # True for better quality, slower inference
)

model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Calibration data — use your actual domain data if possible
calibration_data = [
    "Your domain-specific text here...",
] * 512

model.quantize(calibration_data)
model.save_quantized("./mistral-7b-gptq-4bit-128g")
tokenizer.save_pretrained("./mistral-7b-gptq-4bit-128g")

AWQ via AutoAWQ (requires CUDA, Ampere+):

# quantize_awq.py
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_id = "mistralai/Mistral-7B-v0.1"
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",  # or "GEMV" for batch=1
}

model = AutoAWQForCausalLM.from_pretrained(model_id, **{"low_cpu_mem_usage": True})
tokenizer = AutoTokenizer.from_pretrained(model_id)

calibration_data = ["Your domain text..."] * 512
model.quantize(tokenizer, quant_config=quant_config, calib_data=calibration_data)
model.save_quantized("./mistral-7b-awq-4bit-128g")
tokenizer.save_pretrained("./mistral-7b-awq-4bit-128g")

Option B: Download a verified quant

If you must download, only pull from:

• The model author’s official repo (e.g., TheBloke/, bartowski/, QuantFactory/)
• Repos with gguf or gptq in the name and a README.md documenting the quantization command used
• Never download a quant without a corresponding config.json or quantization metadata

# GGUF: prefer bartowski repos (llama.cpp CI builds)
huggingface-cli download bartowski/Mistral-7B-Instruct-v0.3-GGUF \
  --include "Mistral-7B-Instruct-v0.3-Q4_K_M.gguf" \
  --local-dir ./models

# GPTQ: prefer official AutoGPTQ builds
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.3-GPTQ \
  --include "gptq_model-4bit-128g.safetensors" \
  --local-dir ./models

Verification: Check the file hash against the repo’s stated SHA256. Run llama-gguf-hash model.gguf or safetensors metadata inspection.

Step 6: Load and smoke-test in your target runtime

Each runtime has loading quirks. Verify the model loads, generates, and hits expected token throughput.

Llama.cpp / ollama (GGUF)

# llama.cpp CLI
./llama-cli -m model.q4_k_m.gguf -n 128 -p "The capital of France is" -ngl 99

# ollama (create Modelfile first)
cat > Modelfile <<'EOF'
FROM ./model.q4_k_m.gguf
PARAMETER temperature 0.1
EOF
ollama create mistral-q4 -f Modelfile
ollama run mistral-q4 "The capital of France is"

ExLlamaV2 (GPTQ, fastest CUDA inference)

# test_exllama.py
from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache, ExLlamaV2Tokenizer
from exllamav2.generator import ExLlamaV2StreamingGenerator

config = ExLlamaV2Config()
config.model_dir = "./mistral-7b-gptq-4bit-128g"
config.prepare()

model = ExLlamaV2(config)
cache = ExLlamaV2Cache(model, max_seq_len=4096)
tokenizer = ExLlamaV2Tokenizer(config)

generator = ExLlamaV2StreamingGenerator(model, cache, tokenizer)
generator.warmup()

prompt = "The capital of France is"
ids = tokenizer.encode(prompt)
generator.begin_stream(ids, gen_settings={"temperature": 0.1, "top_p": 0.95})

while True:
    chunk, eos, _ = generator.stream()
    print(chunk, end="", flush=True)
    if eos:
        break
print()

VLLM (GPTQ/AWQ, production serving)

# GPTQ
vllm serve ./mistral-7b-gptq-4bit-128g --quantization gptq --max-model-len 4096

# AWQ
vllm serve ./mistral-7b-awq-4bit-128g --quantization awq --max-model-len 4096

Verification checklist:

• Model loads without OOM or kernel errors
• Generates coherent text for 5+ test prompts
• Token throughput meets your SLA (measure with llama-bench or vLLM’s /metrics)
• No NaN or inf in logits (check with --logits-all in llama.cpp)

Step 7: Profile memory and throughput at your batch size

Single-stream latency ≠ production throughput. Profile at your actual concurrency.

# llama.cpp: measure prompt processing + generation at batch sizes 1, 4, 8
./llama-bench -m model.q4_k_m.gguf -ngl 99 -b 1,4,8 -p 512 -n 128

# vLLM: use built-in benchmark
python -m vllm.entrypoints.benchmark \
  --model ./mistral-7b-gptq-4bit-128g \
  --quantization gptq \
  --batch-size 8 \
  --input-len 512 \
  --output-len 128 \
  --num-prompts 100

Key metrics to capture:

• Prompt processing throughput (tokens/sec during prefill)
• Generation throughput (tokens/sec during decode)
• VRAM peak (via nvidia-smi dmon or tegrastats on Jetson)
• Time to first token (TTFT) at your batch size

If Q4_K_M gives you 45 tok/s at batch 8 but Q5_K_M drops to 32 tok/s with only marginal quality gain, stay at Q4_K_M. The throughput cliff is real — larger quants reduce kernel occupancy.

Step 8: Lock the decision in your deployment pipeline

Don’t re-decide every deploy. Record the exact quantization command, calibration data hash, and eval results in your model registry.

# model-card.yaml (commit alongside model artifact)
model: mistral-7b-instruct-v0.3
format: GGUF
quantization: Q4_K_M
source_commit: a1b2c3d4 (HF model repo)
quant_command: "llama-quantize model.f16.gguf model.q4_k_m.gguf Q4_K_M"
calibration_data_hash: sha256:e3f4...
eval_pass_rate: 0.89
eval_date: "2025-01-15"
runtime: llama.cpp b4567
hardware_tested: ["RTX 4090 24GB", "M2 Max 96GB"]
vram_usage_gb: 5.2
throughput_tok_s: 48 (batch=8)

Automate this in CI. When a new base model drops, the pipeline re-quantizes, re-evals, and either promotes or blocks with a diff report.

Common pitfalls to avoid

Mixing quantization and fine-tuning: Never quantize a LoRA adapter separately and expect it to merge cleanly. Merge adapters to base FP16 first, then quantize the merged model.

Ignoring context length: Some GPTQ/AWQ kernels have hardcoded max sequence limits (often 2048 or 4096). GGUF supports arbitrary context via RoPE scaling. If you need 32K+ context, GGUF is safer.

Assuming INT8 saves you: INT8 (bitsandbytes or GGUF Q8_0) is only ~2x smaller than FP16 but often slower than 4-bit on modern GPUs due to kernel support. Use INT8 only for calibration baselines or when 4-bit genuinely fails quality.

Trusting leaderboard numbers: The LMSYS Chatbot Arena or OpenLLM Leaderboard scores reflect specific quants (usually GPTQ 4bit-128g or GGUF Q4_K_M) on specific prompts. Your domain differs. Run your own eval.

Summary decision tree

START
  ├─ Apple Silicon? → GGUF (llama.cpp / MLX) → Q4_K_M → eval → done
  ├─ AMD / mixed fleet / CPU-only? → GGUF → Q4_K_M → eval → done
  └─ NVIDIA CUDA (Ampere+)?
       ├─ Max throughput, batch > 1? → GPTQ 4bit-128g (ExLlamaV2/vLLM) → eval → done
       ├─ Need AWQ-specific kernel (rare)? → AWQ 4bit-128g → eval → done
       └─ Simplicity / portability matters? → GGUF Q4_K_M → eval → done

The format matters less than the discipline: define your floor, test against real prompts, profile at production batch sizes, and record the provenance. Everything else is noise.