What is GPTQ quantization? How it works

What is GPTQ? GPTQ (Generative Pre-trained Transformer Quantization) is a post-training quantization method that compresses LLM weights to 3–4 bits per parameter by solving a layer-wise weight reconstruction problem using approximate second-order information. Unlike naive rounding, GPTQ minimizes the squared error between the original and quantized layer outputs by accounting for the Hessian of the loss with respect to weights, enabling 4-bit models that retain near-FP16 perplexity.

How GPTQ works

GPTQ operates on one linear layer at a time. For a weight matrix $W \in \mathbb{R}^{d \times n}$ and calibration inputs $X \in \mathbb{R}^{m \times d}$, the goal is to find a quantized matrix $\hat{W}$ that minimizes the output reconstruction error:

$$\min_{\hat{W}} |WX - \hat{W}X|_F^2$$

This is equivalent to minimizing $|(W - \hat{W})X|_F^2$. The key insight: the importance of each weight depends on the input data distribution. GPTQ approximates the Hessian $H = 2XX^\top$ (or a diagonal approximation) to weight the quantization error per output channel.

The algorithm processes columns of $W$ sequentially (output channels). For each column $w_i$:

1. Quantize greedily: Round $w_i$ to the nearest grid point in the target bitwidth (e.g., INT4).
2. Compute error: $\Delta = w_i - \hat{w}_i$.
3. Propagate correction: Update remaining unquantized columns $j > i$ by subtracting $\Delta \cdot H_{ij} / H_{ii}$, where $H$ is the (approximate) Hessian. This compensates for the quantization error in downstream columns.
4. Repeat until all columns are quantized.

The original paper uses a diagonal Hessian approximation $H \approx \text{diag}(2XX^\top)$ for memory efficiency, plus a “dampening” factor (typically 0.01) added to the diagonal for numerical stability. The calibration set is typically 128–512 sequences of 2048 tokens drawn from the training distribution (e.g., C4, WikiText).

# Simplified GPTQ per-layer logic (conceptual, not production code)
def gptq_quantize_layer(W, X, bits=4, group_size=128, damp=0.01):
    """
    W: [out_features, in_features] float16
    X: [num_samples, in_features] float16 calibration inputs
    """
    H = 2 * (X.T @ X)  # [in_features, in_features]
    H += damp * torch.eye(H.shape[0], device=H.device)
    
    # Cholesky for efficient solves
    L = torch.linalg.cholesky(H)
    Hinv = torch.cholesky_inverse(L)
    
    Q = torch.zeros_like(W, dtype=torch.int8)
    scales = torch.zeros(W.shape[0], dtype=torch.float16)
    zeros = torch.zeros(W.shape[0], dtype=torch.float16)
    
    for i in range(W.shape[0]):  # each output channel
        w = W[i].float()
        
        # Quantize with per-group scaling
        for g in range(0, w.shape[0], group_size):
            w_g = w[g:g+group_size]
            scale = (w_g.max() - w_g.min()) / (2**bits - 1)
            zero = -w_g.min() / scale
            q_g = torch.round(w_g / scale + zero).clamp(0, 2**bits - 1)
            
            Q[i, g:g+group_size] = q_g.to(torch.int8)
            scales[i] = scale
            zeros[i] = zero
        
        # Error propagation to remaining rows
        error = (W[i] - dequantize(Q[i], scales[i], zeros[i])).float()
        if i + 1 < W.shape[0]:
            W[i+1:] -= error.unsqueeze(0) @ (Hinv[i, i+1:] / Hinv[i, i]).unsqueeze(1)
    
    return Q, scales, zeros

Production implementations (AutoGPTQ, ExLlamaV2, vLLM’s GPTQ kernel) add critical optimizations:

• Group-wise quantization: Scales and zero-points per 128 or 64 weights (not per-channel) to handle outliers.
• Act-order: Reorder input channels by Hessian diagonal magnitude before quantization; restores original order after. Reduces error significantly.
• Triton/CUDA kernels: Fused dequantize+matmul for inference speed.
• Descending activation order: Process columns in order of decreasing Hessian diagonal, which empirically lowers reconstruction error.

Why GPTQ matters

GPTQ was the first method to make 4-bit LLM inference practical on consumer GPUs without retraining. Before GPTQ (late 2022), the options were:

• 8-bit quantization (LLM.int8()): 2× compression, minimal quality loss, but still VRAM-heavy for 7B+ models.
• Naive 4-bit rounding: Catastrophic perplexity degradation (>10× PPL increase).
• QLoRA / fine-tuning: Requires GPU-hours per model, not a drop-in compression step.

GPTQ changed the economics: a 7B parameter model drops from ~14 GB (FP16) to ~4.5 GB (INT4, group_size=128), fitting on a 24 GB consumer card with headroom for KV cache and batch >1. A 70B model drops from ~140 GB to ~42 GB, moving from “requires 8×A100” to “runs on 2×A100 80GB or 4×3090/4090.”

The quality preservation is real. On benchmarks like MMLU, GSM8K, and HumanEval, GPTQ-INT4 models typically score within 1–3% absolute of their FP16 baselines. Perplexity on WikiText-2 increases by roughly 0.1–0.3 points for 7B–70B models — negligible for most downstream tasks.

# Typical VRAM footprint comparison (weights only, no KV cache)
# Model          FP16 (GB)   GPTQ-4bit (GB)   Reduction
# Llama-2-7B       13.5           3.8            3.5×
# Llama-2-13B      26.0           7.3            3.6×
# Llama-2-70B      138.0          38.5           3.6×
# Mixtral-8x7B     90.0           26.0           3.5×

Inference latency also improves on memory-bandwidth-bound hardware (most consumer GPUs). The compute is still FP16 matmul (dequantize on the fly), but weight fetch bandwidth drops 4×. On an RTX 4090, a GPTQ-4bit 7B model can hit 80–100 tok/s single-batch vs 40–50 tok/s for FP16.

Concrete example: quantizing Llama-3-8B with AutoGPTQ

Here’s a complete, runnable workflow. You need a calibration dataset — use a slice of the training data or a representative domain corpus.

# Install
pip install auto-gptq optimum[exporters] datasets accelerate

# Quantize
python -m auto_gptq.cli.quantize \
  --model_id meta-llama/Meta-Llama-3-8B \
  --quant_path ./Llama-3-8B-GPTQ-4bit \
  --bits 4 \
  --group_size 128 \
  --desc_act \
  --dataset c4 \
  --num_samples 512 \
  --seq_len 2048 \
  --device cuda:0

Key flags explained:

• --bits 4: Target INT4. --bits 3 works but quality drops sharply below 4 bits for most models.
• --group_size 128: Per-group scales/zeros. Smaller groups (64, 32) improve quality at cost of metadata overhead (~0.5% size per halving). 128 is the sweet spot.
• --desc_act: Enable act-order (descending activation). Always use this. It costs zero inference overhead and recovers ~0.1–0.2 perplexity points.
• --dataset c4 --num_samples 512 --seq_len 2048: Calibration set. 512×2048 tokens ≈ 1M tokens, takes 10–20 minutes on an A100. More samples help marginally; 128 is the practical minimum.
• --device cuda:0: Quantization runs on GPU. Requires ~24 GB for 8B model (FP16 weights + Hessian + calibration activations).

Output structure:

Llama-3-8B-GPTQ-4bit/
├── config.json              # quantization_config with bits, group_size, desc_act
├── model.safetensors        # quantized weights (int8 packed) + scales + zeros
├── quantize_config.json     # mirrors CLI args for reproducibility
└── tokenizer files...

Load and run with any GPTQ-compatible engine:

from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer

model = AutoGPTQForCausalLM.from_quantized(
    "./Llama-3-8B-GPTQ-4bit",
    device="cuda:0",
    use_triton=True,      # faster kernels on Ampere+
    inject_fused_attention=True,
    use_cuda_fp16=True,   # FP16 compute
)
tokenizer = AutoTokenizer.from_pretrained("./Llama-3-8B-GPTQ-4bit")

inputs = tokenizer("The future of quantization is", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For production serving, use vLLM or TGI with the GPTQ backend — they handle continuous batching, paged attention, and kernel selection automatically.

# vLLM OpenAI-compatible server
vllm serve ./Llama-3-8B-GPTQ-4bit \
  --quantization gptq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --tensor-parallel-size 1

Common misconceptions

“GPTQ requires retraining or fine-tuning”

False. GPTQ is strictly post-training. The calibration forward passes collect activation statistics; no gradients, no optimizer steps, no label data. It takes 10–30 minutes on one GPU, not hours.

“GPTQ-4bit is always better than AWQ-4bit”

Not universally. AWQ (Activation-aware Weight Quantization) protects salient weights using activation magnitude instead of Hessian. AWQ is faster to quantize (no Hessian computation) and sometimes edges out GPTQ on instruction-tuned models with heavy outlier channels. GPTQ tends to win on base models and code tasks. Benchmark both on your model family.

“Group size doesn’t matter much”

Group size is the single most impactful hyperparameter after bitwidth.

• group_size=128: Standard, good quality/size trade-off.
• group_size=64: +0.5–1% model size, measurable quality gain on 70B+ models.
• group_size=32: Diminishing returns, metadata overhead ~1.5%.
• group_size=-1 (per-channel): Smallest size, but outlier channels destroy quality. Avoid.

“You can quantize any layer”

Embedding layers and the final LM head are typically kept in FP16 (or BF16). Quantizing them hurts quality disproportionately because:

• Embeddings: Every token passes through; error accumulates across sequence length.
• LM head: Directly produces logits; quantization noise maps to probability distortion.

AutoGPTQ and vLLM exclude these by default. If you force-quantize them, expect 2–5× perplexity degradation.

“GPTQ works equally well at 3-bit and 2-bit”

4-bit is the practical floor for general-purpose LLMs without accuracy recovery techniques. At 3-bit (INT3), expect 5–15% relative perplexity increase and noticeable reasoning degradation. At 2-bit, models largely collapse unless you use:

• Mixed precision: Keep sensitive layers (attention out-proj, FFN down-proj) at 4-bit.
• Quantization-aware training (QAT): Fine-tune with fake quantization.
• Higher group resolution: group_size=32 or 16 with per-group scales.

For 3-bit, AWQ or QuIP# often outperform GPTQ.

“The calibration dataset must match your exact domain”

Calibration data should cover the activation distribution the model sees at inference. For general chat/code, C4 or Pile works fine. For narrow domains (legal, medical, specific codebases), using 512–1024 samples from that domain can recover 0.05–0.15 perplexity points. But mismatched calibration (e.g., English-only for a multilingual model) hurts more than small sample size.

When to use GPTQ vs alternatives

Practical tips from production

1. Always verify perplexity on a held-out slice of your calibration data before deploying. A 0.5+ PPL increase over FP16 baseline signals a quantization bug (wrong group_size, missing desc_act, embeddings quantized).

2. Test generation quality, not just perplexity. Some quantized models pass PPL checks but hallucinate more on long-context or structured output tasks. Run your actual eval suite.

3. Save the quantization config. The quantize_config.json is your reproducibility artifact. Without it, you cannot re-quantize identically when the base model updates.

4. Watch for kernel compatibility. use_triton=True requires Ampere (RTX 30-series) or newer. On Turing (T4, RTX 20-series), fall back to CUDA kernels — slower but functional.

5. Don’t quantize MoE models naively. For Mixtral, DeepSeek-MoE, etc., quantize each expert independently with its own calibration pass, or use a method that accounts for router activations (AWQ handles this better out of the box).

6. Monitor VRAM at runtime. GPTQ weights are dequantized on-the-fly to FP16 for matmul. Peak VRAM = quantized weights + dequantization buffer (one layer at a time) + KV cache + activations. On 24 GB cards, 34B-4bit fits with ~4K context; 70B-4bit needs tensor parallel or offloading.

GPTQ remains the workhorse of LLM compression because it hits the sweet spot: no training, broad hardware support, predictable quality, and mature tooling. If you’re shipping a quantized model today, GPTQ-4bit with act-order is still the default choice — until you have a measured reason to switch.