What is quantization in AI? A plain-English guide

Quantization in AI is the process of reducing the numerical precision of a model’s weights and activations — typically from 32-bit floating point (FP32) down to 8-bit integers (INT8) or 4-bit integers (INT4) — to shrink model size and accelerate inference. The core idea is simple: neural networks tolerate surprisingly aggressive rounding because their learned representations are inherently redundant and noise-tolerant. Done well, quantization delivers 2-4x memory savings and throughput gains with negligible quality loss; done poorly, it collapses model capability.

How quantization works

At inference time, every matrix multiply in a transformer looks like y = x @ W^T where x is your activation vector and W is a weight matrix. In FP32, each element occupies 4 bytes. In INT8, each occupies 1 byte. The quantization process maps a continuous floating-point range onto a discrete integer grid.

Symmetric vs asymmetric quantization

Symmetric quantization maps [-max, max] to [-127, 127] (or [-128, 127] for INT8) with a single scale factor:

def quantize_symmetric(x: np.ndarray, bits: int = 8) -> tuple[np.ndarray, float]:
    qmax = 2**(bits - 1) - 1
    scale = x.abs().max() / qmax
    qx = (x / scale).round().clamp(-qmax, qmax).to(torch.int8)
    return qx, scale.item()

def dequantize_symmetric(qx: np.ndarray, scale: float) -> np.ndarray:
    return qx.float() * scale

Asymmetric quantization maps [min, max] to [0, 255] (for UINT8) using both scale and zero-point:

def quantize_asymmetric(x: np.ndarray, bits: int = 8) -> tuple[np.ndarray, float, int]:
    qmin, qmax = 0, 2**bits - 1
    scale = (x.max() - x.min()) / (qmax - qmin)
    zero_point = qmin - (x.min() / scale).round()
    qx = (x / scale + zero_point).round().clamp(qmin, qmax).to(torch.uint8)
    return qx, scale.item(), zero_point.item()

Symmetric is simpler and hardware-friendly (no zero-point arithmetic in the hot path). Asymmetric handles skewed distributions better — ReLU activations, for instance, are strictly non-negative.

Per-tensor vs per-channel vs group quantization

Per-tensor uses one scale for the entire weight matrix. Fast, minimal metadata, but hurts accuracy on outlier channels.

Per-channel computes a scale per output channel (row of W). Standard for weight-only quantization on GPUs; the extra scales are negligible overhead.

Group quantization (e.g., GPTQ, AWQ) partitions each row into groups of 64 or 128 weights, each with its own scale. This captures intra-channel variation and is the default for 4-bit LLM quantization.

# Group quantization pseudo-code
def quantize_grouped(W: torch.Tensor, group_size: int = 128, bits: int = 4):
    assert W.dim() == 2
    out_channels, in_channels = W.shape
    assert in_channels % group_size == 0
    
    W_grouped = W.view(out_channels, -1, group_size)
    scales = W_grouped.abs().amax(dim=-1) / (2**(bits - 1) - 1)
    qW = (W_grouped / scales.unsqueeze(-1)).round().clamp_(-2**(bits-1), 2**(bits-1)-1)
    return qW.to(torch.int8), scales

Why quantization matters for inference

Three constraints drive quantization adoption: memory bandwidth, compute throughput, and model footprint.

Memory bandwidth is the bottleneck

Modern GPUs spend most cycles waiting on data. An A100 80GB delivers ~1.5 TB/s memory bandwidth but 312 TFLOPS of FP16 compute. For a 7B parameter model at FP16 (14 GB weights), a single forward pass reads ~14 GB from VRAM. At INT4 (3.5 GB), you read 4x less data — directly translating to higher tokens/second when bandwidth-bound.

# Rough bandwidth math for 7B model, batch=1, seq=2048
# FP16: 14 GB weights + 2 GB KV cache ≈ 16 GB read per token
# INT4:  3.5 GB weights + 2 GB KV cache ≈ 5.5 GB read per token
# Bandwidth-limited throughput scales ~3x

Compute throughput on integer tensor cores

Hopper (H100) and Blackwell (B200) tensor cores natively accelerate INT8 and INT4 matrix multiplies. The theoretical peak for INT8 is 2x FP16; for INT4 it’s 4x. Realized speedups are lower due to dequantization overhead and kernel launch latency, but 1.5-2x throughput gains are typical for weight-only quantization.

Model footprint determines deployability

Quantization is what makes 70B models run on a single 80GB GPU, and 7B models run on a laptop.

Post-training quantization (PTQ) vs quantization-aware training (QAT)

Post-training quantization

PTQ takes a trained FP16/BF16 model and quantizes weights (and optionally activations) without retraining. It’s fast — minutes to hours — and works well down to INT8. Below INT8, naive PTQ degrades perplexity sharply.

Weight-only PTQ (W8A16, W4A16): Quantize weights, keep activations in FP16. Minimal accuracy loss at INT8; 1-2% perplexity increase at INT4 on 7B models.

Weight + activation PTQ (W8A8, W4A4): Quantize both. Requires calibration data (typically 128-512 sequences) to compute activation scales. Hardware support for INT4 activations is limited; most “W4A4” implementations actually dequantize to FP16 for the matmul.

# Calibration for activation quantization
def calibrate(model, dataloader, num_batches=128):
    model.eval()
    act_stats = defaultdict(list)
    
    def hook(name):
        def fn(module, inp, out):
            act_stats[name].append(out.detach().abs().amax(dim=(0,1)))
        return fn
    
    handles = []
    for name, module in model.named_modules():
        if isinstance(module, (nn.Linear, nn.Conv2d)):
            handles.append(module.register_forward_hook(hook(name)))
    
    with torch.no_grad():
        for i, batch in enumerate(dataloader):
            if i >= num_batches:
                break
            model(batch)
    
    for h in handles:
        h.remove()
    
    return {k: torch.stack(v).max(dim=0).values for k, v in act_stats.items()}

Quantization-aware training

QAT simulates quantization during training using straight-through estimators (STE) for the rounding operation. The forward pass quantizes; the backward pass passes gradients through as if quantization were identity.

class FakeQuantize(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, scale, zero_point, qmin, qmax):
        qx = (x / scale + zero_point).round().clamp(qmin, qmax)
        return qx * scale - zero_point * scale  # dequantize for next op
    
    @staticmethod
    def backward(ctx, grad_output):
        return grad_output, None, None, None, None  # STE

QAT recovers most INT4 accuracy loss but requires full training infrastructure, GPU-hours, and careful learning rate scheduling. For most teams, PTQ with advanced algorithms (GPTQ, AWQ, HQQ) is the pragmatic choice.

Advanced PTQ algorithms: GPTQ, AWQ, HQQ

GPTQ (Generalized Post-Training Quantization)

GPTQ quantizes layer-by-layer, solving a local optimization problem per layer using second-order information (the Hessian of the loss w.r.t weights). It processes columns sequentially, updating remaining weights to compensate for quantization error.

# GPTQ core update (simplified)
def gptq_quantize_layer(W, H, bits=4, group_size=128):
    # W: [out_features, in_features], H: [in_features, in_features] (Hessian)
    # Process in groups
    Q = torch.zeros_like(W, dtype=torch.int8)
    for i in range(0, W.shape[1], group_size):
        W_g = W[:, i:i+group_size]
        H_g = H[i:i+group_size, i:i+group_size]
        
        # Cholesky of Hessian for numerical stability
        L = torch.linalg.cholesky(H_g + 1e-6 * torch.eye(group_size))
        
        # Quantize column by column
        for j in range(group_size):
            w_col = W_g[:, j]
            # Optimal quantized value minimizing local error
            q_col = quantize_column(w_col, bits)
            Q[:, i+j] = q_col
            
            # Update remaining weights: W -= (w - q) * H_g[j] / H_g[j,j]
            err = (w_col - dequantize(q_col)).unsqueeze(1)
            W_g[:, j+1:] -= err @ (L[j, j+1:].unsqueeze(0) / L[j, j])
    
    return Q

GPTQ takes ~4 GPU-hours for a 70B model at INT4. Quality is excellent — often within 0.1-0.3 perplexity points of FP16.

AWQ (Activation-aware Weight Quantization)

AWQ observes that not all weights matter equally: weights multiplied by large-magnitude activations contribute more to output error. It scales weight channels inversely to activation magnitudes before quantization, then corrects the scale post-quantization.

# AWQ scaling (conceptual)
def awq_scale_search(W, act_scales, bits=4, n_grid=20):
    # act_scales: [in_features] - per-channel activation max
    best_loss = float('inf')
    best_scales = None
    
    for s in np.linspace(0, 1, n_grid):
        # Scale weights: W' = W * (act_scales^s)
        # Scale activations: x' = x / (act_scales^s)
        scaled_W = W * (act_scales ** s).unsqueeze(0)
        qW = quantize(scaled_W, bits)
        deqW = dequantize(qW) / (act_scales ** s).unsqueeze(0)
        
        loss = (W - deqW).pow(2).mean()
        if loss < best_loss:
            best_loss = loss
            best_scales = act_scales ** s
    
    return best_scales

AWQ is faster than GPTQ (minutes vs hours) and requires only calibration data, no gradient computation. It’s the default for many open-weight model releases.

HQQ (Half-Quadratic Quantization)

HQQ formulates quantization as a half-quadratic optimization problem, solvable via alternating minimization. It’s extremely fast (seconds per layer) and competitive with GPTQ/AWQ, with a cleaner mathematical foundation.

A concrete example: quantizing Llama-3-8B

# Using llama.cpp (GGUF format, k-quantization)
# Download FP16 model
huggingface-cli download meta-llama/Meta-Llama-3-8B --local-dir llama3-8b-fp16

# Quantize to Q4_K_M (4-bit, mixed precision, medium quality)
llama-quantize llama3-8b-fp16 llama3-8b-q4_k_m.gguf Q4_K_M

# Result: ~4.7 GB vs 16 GB FP16
# Perplexity on WikiText-2: FP16 ~5.8, Q4_K_M ~6.1
# Throughput on M2 MacBook Pro: ~45 tok/s vs ~12 tok/s (FP16)

# Using transformers + bitsandbytes (4-bit NF4)
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",        # NormalFloat4, better than INT4 for normal-ish weights
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,   # Quantize the quantization constants
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# VRAM usage: ~6.5 GB (vs 16 GB FP16)
# Throughput on A10G: ~120 tok/s (vs ~60 tok/s FP16)

NF4 (NormalFloat4) uses a non-uniform quantization grid optimized for zero-centered normal distributions — a better fit for transformer weights than uniform INT4.

Common misconceptions

“Quantization always degrades quality”

At INT8, quality loss is typically unmeasurable on downstream tasks. At INT4, the gap is real but often acceptable for chat, summarization, and coding. The degradation is not uniform: creative writing suffers more than fact-seeking QA; long-context reasoning degrades faster than short prompts.

“You can quantize activations to INT4 for free”

Most “W4A4” implementations dequantize activations to FP16/BF16 before the matmul because:

1. No consumer GPU has native INT4 tensor core accumulation
2. Accumulation in INT32 requires 32-bit registers, reducing occupancy
3. Dequantization overhead often exceeds the compute savings

True INT4 activation kernels exist (CUTLASS, custom PTX) but are model-specific and fragile.

“Quantization is a one-time choice”

Quantization interacts with:

• Prompt length: Longer contexts amplify activation quantization error
• Temperature: Higher temperature exposes weight quantization noise
• Fine-tuning: A quantized base model fine-tuned with LoRA often recovers INT4 losses
• Distillation: Quantized teacher → FP16 student can transfer quality

“Smaller bits are always better”

INT3 and INT2 exist but require:

• Group size ≤ 32 (more metadata, less compute density)
• Per-group scale + zero-point (2-3 bytes per group)
• Aggressive outlier handling (separate FP16 storage for 0.1% of weights)

The Pareto frontier for most LLMs today is INT4 with group size 128. INT3 saves ~25% more memory but costs 2-3x perplexity degradation.

“Calibration data doesn’t matter”

For activation quantization and AWQ, calibration data must match your deployment distribution. Using WikiText-2 to calibrate a code model shifts activation scales wrong, causing 10-20% throughput collapse from dynamic range clipping. Use 512 sequences from your actual traffic.

Practical recommendations

Start with weight-only INT4 (NF4 or INT4 group-128). Use bitsandbytes for PyTorch, llama.cpp for CPU/Apple Silicon, or vLLM/TensorRT-LLM for production GPU serving. Skip activation quantization until you’ve proven weight-only isn’t sufficient.

Profile before committing. Quantization changes the bottleneck. A model that was compute-bound at FP16 may become latency-bound at INT4 due to kernel launch overhead. Measure:

• Time-to-first-token (TTFT) at various batch sizes
• Inter-token latency (decode throughput)
• Peak VRAM including KV cache

Keep an FP16 reference. Run your eval suite against both quantized and FP16. Track perplexity, task-specific metrics (HumanEval, MMLU, your internal benchmarks), and qualitative “vibe” on open-ended prompts.

Consider mixed precision. Critical layers (embedding, output projection, first/last few transformer blocks) often stay FP16 while middle layers quantize to INT4. This recovers 50-80% of the quality gap for ~10% memory cost.

When not to quantize

• Training or fine-tuning (use BF16/FP32 for stability)
• Models < 1B params (overhead exceeds savings)
• Latency-critical paths where kernel launch variance matters
• When you need exact reproducibility (quantization introduces non-determinism from rounding)

Quantization is a deployment optimization, not a modeling one. Apply it at the serving layer, not the research layer.