How quantization shrinks weights without retraining

Quantization compresses model weights from high-precision floats to lower-bit integers, shrinking memory footprint and accelerating inference without retraining. The core idea is straightforward: map FP16 or FP32 values to INT8, INT4, or even INT2 representations using calibration data, then run inference with integer arithmetic. This guide walks through the main post-training quantization (PTQ) methods, when to use each, and the pitfalls that catch engineers off guard.

Why quantize without retraining

Retraining or fine-tuning a quantized model (quantization-aware training, or QAT) typically recovers more accuracy, but it requires GPU hours, labeled data, and pipeline changes. Post-training quantization skips all that. You take an existing checkpoint, run a calibration pass over a few hundred samples, and produce a quantized artifact ready for deployment. The tradeoff is a small accuracy drop — usually 0.5–2% on downstream tasks for 4-bit weights — but the operational simplicity makes PTQ the default choice for most teams shipping open-weight models.

The memory savings are dramatic. A 7B parameter model at FP16 needs ~14 GB VRAM. At 4-bit, it fits in ~4.5 GB. At 3-bit, ~3.5 GB. This difference determines whether you can serve the model on a single consumer GPU or need multi-GPU setups. For latency-bound workloads, integer matrix multiplication on modern GPUs (via Tensor Cores) or CPUs (via AVX2/NEON) often yields 2–4x throughput gains over FP16.

The quantization pipeline

Every PTQ method follows the same skeleton:

1. Collect calibration data — a representative sample of your input distribution (typically 128–1024 sequences).
2. Compute quantization parameters — per-tensor or per-channel scale and zero-point for each weight matrix.
3. Quantize weights — apply the mapping, producing integer tensors and metadata.
4. Validate — run evaluation benchmarks to measure quality regression.
5. Export — serialize in a format your inference engine expects (GGUF, Safetensors, ONNX, etc.).

The differences between methods lie in step 2: how they compute scales, whether they handle outliers, and whether they preserve important weights at higher precision.

Basic per-tensor symmetric quantization

The simplest approach uses a single scale per weight tensor, symmetric around zero (no zero-point). For a weight matrix W in FP16:

import torch

def quantize_per_tensor_symmetric(W: torch.Tensor, bits: int = 8) -> tuple[torch.Tensor, float]:
    """
    Symmetric per-tensor quantization to signed int.
    Returns (quantized_weights, scale).
    """
    assert W.dtype in (torch.float16, torch.float32, torch.bfloat16)
    max_val = W.abs().max().item()
    qmax = 2 ** (bits - 1) - 1
    scale = max_val / qmax
    W_q = torch.round(W / scale).clamp(-qmax, qmax).to(torch.int8 if bits == 8 else torch.int32)
    return W_q, scale

def dequantize(W_q: torch.Tensor, scale: float) -> torch.Tensor:
    return W_q.to(torch.float16) * scale

This works adequately for 8-bit. At 4-bit, the coarse granularity of a single scale across an entire layer causes noticeable degradation, especially in attention projection matrices where outlier channels dominate the dynamic range.

Per-channel quantization handles outliers

Transformer weight matrices have highly non-uniform channel distributions. A few channels in q_proj or v_proj can have magnitudes 10–100x larger than the median. Per-channel quantization computes a separate scale for each output channel (row of the weight matrix), dramatically reducing clipping error.

def quantize_per_channel_symmetric(W: torch.Tensor, bits: int = 4) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Per-output-channel symmetric quantization.
    W shape: [out_features, in_features]
    Returns (quantized_weights, scales) where scales shape: [out_features]
    """
    assert W.dim() == 2
    qmax = 2 ** (bits - 1) - 1
    max_per_channel = W.abs().amax(dim=1)  # [out_features]
    scales = max_per_channel / qmax
    # Broadcast scales to [out_features, 1] for division
    W_q = torch.round(W / scales.unsqueeze(1)).clamp(-qmax, qmax)
    return W_q.to(torch.int8 if bits <= 8 else torch.int32), scales

Per-channel is the baseline for any serious 4-bit quantization. Most inference engines (llama.cpp, vLLM, TensorRT-LLM) expect per-channel scales for linear layers.

GPTQ: one-shot weight reconstruction

GPTQ (Generalized Post-Training Quantization) improves on basic PTQ by reconstructing each weight row to minimize the layer-wise reconstruction error. It processes columns sequentially, updating the remaining unquantized weights to compensate for quantization error already introduced. The algorithm is derived from Optimal Brain Quantization and uses the inverse Hessian of the layer’s input activations.

# Simplified GPTQ core loop (educational only; use auto-gptq or optimum in practice)
def gptq_quantize_layer(W: torch.Tensor, H_inv: torch.Tensor, bits: int = 4, group_size: int = 128):
    """
    W: [out_features, in_features] float weight
    H_inv: [in_features, in_features] inverse Hessian (X^T X)^-1
    group_size: quantization group size along input dimension
    """
    out_features, in_features = W.shape
    W_q = torch.zeros_like(W, dtype=torch.int32)
    scales = torch.zeros(out_features, in_features // group_size, dtype=torch.float16)
    zeros = torch.zeros_like(scales)

    for i in range(0, in_features, group_size):
        j = min(i + group_size, in_features)
        w_group = W[:, i:j].clone().float()
        h_inv_group = H_inv[i:j, i:j]

        # Quantize group
        max_val = w_group.abs().amax(dim=1, keepdim=True)
        qmax = 2 ** (bits - 1) - 1
        scale = max_val / qmax
        zero = torch.zeros_like(scale)
        w_q_group = torch.round(w_group / scale).clamp(-qmax, qmax).to(torch.int32)

        W_q[:, i:j] = w_q_group
        scales[:, i // group_size] = scale.squeeze()
        zeros[:, i // group_size] = zero.squeeze()

        # Reconstruction error compensation
        err = (w_q_group * scale - w_group) @ h_inv_group
        W[:, j:] -= err @ H_inv[j:, i:j]

    return W_q, scales, zeros

In practice, you don’t implement this yourself. Use auto-gptq or Hugging Face optimum:

pip install auto-gptq optimum[gptq]

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_id = "meta-llama/Llama-2-7b-hf"
quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False,  # True for act-order GPTQ, slower quantization but better quality
)

model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config=quantize_config, device_map="auto")
calibration_data = ["your calibration text here"] * 512  # or load from dataset
model.quantize(calibration_data)
model.save_quantized("./llama-2-7b-gptq-4bit")

GPTQ at 4-bit with group_size=128 typically matches FP16 within 0.5–1% perplexity. The desc_act=True variant (act-order) reorders columns by activation magnitude before quantization, yielding another ~0.2% improvement at the cost of 2–3x slower quantization time.

AWQ: activation-aware weight quantization

AWQ observes that not all weights matter equally. Channels with large activation magnitudes contribute disproportionately to output error. AWQ searches for a per-channel scaling factor s that minimizes quantization error on important channels, then absorbs s into the weights and compensates by scaling the next layer’s inputs by 1/s.

# Conceptual AWQ scaling (simplified)
def awq_scale_search(W: torch.Tensor, act_scales: torch.Tensor, grid: int = 20):
    """
    W: [out_features, in_features]
    act_scales: [in_features] - average activation magnitude per input channel
    Returns per-channel scale factors s for W
    """
    best_scales = torch.ones(W.shape[1])
    for i in range(W.shape[1]):
        # Search s in [0.01, 1.0] that minimizes weighted quantization error
        # Weight by act_scales[i] - higher activation = more important
        errors = []
        for s in torch.linspace(0.01, 1.0, grid):
            W_scaled = W[:, i] * s
            # Quantize and measure error weighted by act_scales[i]
            # ... quantization logic ...
            errors.append(weighted_error)
        best_scales[i] = torch.linspace(0.01, 1.0, grid)[torch.argmin(torch.tensor(errors))]
    return best_scales

The key insight: scaling weights by s and the next layer’s input by 1/s preserves the mathematical equivalence of the network while redistributing quantization error away from salient channels. AWQ requires no Hessian computation, making it 10–100x faster than GPTQ for quantization, with comparable quality.

pip install awq

from awq import AutoAWQForCausalLM

model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", device_map="auto")
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}
model.quantize(tokenizer, calib_data="pileval", quant_config=quant_config)
model.save_quantized("./llama-2-7b-awq-4bit")

AWQ is the default choice for many open-weight 4-bit releases on Hugging Face (look for -AWQ suffix). It works well with vLLM and TensorRT-LLM kernels.

GGUF and llama.cpp: quantization for CPU and edge

GGUF (GPT-GGML Unified Format) is the container format used by llama.cpp. It supports a zoo of quantization schemes: q4_0, q4_k_m, q5_k_m, q8_0, etc. The k-variants (k-quant) use mixed precision: important weights stay at higher bit-width while others drop lower. This is effectively a hand-tuned version of the saliency ideas in AWQ, baked into the format.

# Convert HF model to GGUF, then quantize
pip install gguf
python convert-hf-to-gguf.py meta-llama/Llama-2-7b-hf --outfile llama-2-7b.f16.gguf
./llama-quantize llama-2-7b.f16.gguf llama-2-7b.q4_k_m.gguf q4_k_m

Common GGUF quantization types:

The k_m variants use a 2-bit super-block scale and 4/6/8-bit sub-block quantization. They’re tuned empirically for transformer architectures. For most engineers deploying on CPU or Apple Silicon, q4_k_m is the starting point.

Calibration data: the silent quality lever

All PTQ methods need calibration data. The quality of your quantized model depends heavily on how well this data matches your production input distribution. Common mistakes:

• Using random tokens or Wikipedia text when your traffic is code, chat, or structured extraction.
• Using too few samples (< 128 sequences) — Hessian estimation becomes noisy.
• Using sequences that exceed model context length — truncation changes activation statistics.

def prepare_calibration_data(tokenizer, dataset_name: str, num_samples: int = 512, seq_len: int = 2048):
    from datasets import load_dataset
    ds = load_dataset(dataset_name, split="train", streaming=True)
    texts = []
    for sample in ds.take(num_samples * 2):  # oversample, filter length
        text = sample["text"] if "text" in sample else sample["content"]
        if len(text) > 100:
            texts.append(text)
            if len(texts) >= num_samples:
                break
    
    # Tokenize and chunk
    encodings = tokenizer(texts, return_tensors="pt", padding=True, truncation=True, max_length=seq_len)
    return encodings.input_ids

For code models, use codeparrot/github-code or your own repos. For chat, use HuggingFaceH4/ultrachat_200k. For domain-specific deployment, sample from your actual request logs (scrubbed for PII).

Common pitfalls and how to detect them

Accuracy regression on long context

Quantization error accumulates across layers. A 0.5% perplexity increase per layer compounds to significant degradation at 32k context. Test with needle-in-haystack or long-context QA benchmarks, not just perplexity on 2k sequences.

KV cache quantization mismatch

If you quantize weights to 4-bit but keep KV cache in FP16, you get memory savings only on weights. For full benefit, quantize KV cache to INT8 or FP8 (supported on H100/Ada). llama.cpp does this automatically with -c k q8_0 style flags. vLLM supports FP8 KV cache via --kv-cache-dtype fp8.

Activation outliers at inference time

Calibration captures training-time activation distributions. If production inputs have heavier tails (e.g., longer sequences, different vocabulary), dynamic range shifts cause clipping. Monitor activation max values in production and re-quantize if they drift > 20% from calibration.

Group size too large for small layers

Group size 128 is standard, but for layers with < 256 input features (e.g., small MoE experts, adapter layers), per-group quantization degrades to near per-tensor. Use group_size = min(128, in_features // 2) or fall back to per-channel for small layers.

Silently broken quantization config

Always verify the quantized model loads and runs a forward pass before committing artifacts:

def sanity_check_quantized(model_path: str, tokenizer):
    from transformers import AutoModelForCausalLM
    import torch
    
    model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto", torch_dtype=torch.float16)
    input_ids = tokenizer("Hello, world!", return_tensors="pt").input_ids.to(model.device)
    with torch.no_grad():
        logits = model(input_ids).logits
    assert not torch.isnan(logits).any(), "NaN logits detected"
    assert not torch.isinf(logits).any(), "Inf logits detected"
    print(f"Sanity check passed. Logits range: [{logits.min():.2f}, {logits.max():.2f}]")

Choosing a method: decision flowchart

START: Need to quantize a model without retraining?
│
├─ Target: CPU / Apple Silicon / edge device
│   └─ Use GGUF q4_k_m via llama.cpp
│
├─ Target: NVIDIA GPU, need maximum throughput
│   ├─ Have 30+ min for quantization, want best 4-bit quality
│   │   └─ GPTQ (desc_act=True, group_size=128)
│   ├─ Need fast quantization (< 5 min), good quality
│   │   └─ AWQ (w_bit=4, group_size=128)
│   └─ Need FP8 / INT8 for H100 / Blackwell
│       └─ TensorRT-LLM PTQ or vLLM FP8 quantization
│
├─ Target: Mixed deployment (GPU + CPU)
│   └─ Quantize to both: AWQ for GPU, GGUF for CPU
│
└─ Model: MoE or mixture-of-experts
    └─ AWQ handles expert routing weights better; avoid GPTQ on router logits

Integration with inference engines

Each engine expects specific artifacts:

When serving multiple quantized variants, tag artifacts clearly: model-awq-4bit-g128, model-gptq-4bit-actorder, model-gguf-q4km. Include quantization config in model cards so downstream consumers can reproduce or debug.

Summary checklist

• Choose quantization method matching your hardware target (GPU → AWQ/GPTQ, CPU → GGUF).
• Prepare calibration data matching production distribution (512+ samples, correct domain).
• Run quantization with appropriate group size (128 default, smaller for narrow layers).
• Validate on perplexity + task-specific evals (not just perplexity).
• Test long-context behavior if context > 4k.
• Verify KV cache quantization strategy aligns with weight quantization.
• Sanity-check loaded model for NaN/Inf logits.
• Export in the format your inference engine expects.
• Tag and version artifacts with quantization config.

Quantization without retraining is a solved engineering problem for 4-bit and above. The remaining challenges are operational: calibration data drift, kernel compatibility across engine versions, and monitoring quality in production. Treat your quantization pipeline like any other build step — versioned, reproducible, and tested.