What is quantization and how it reduces GPU memory use

Quantization is the process of reducing the numerical precision of a model’s weights and activations — typically from 16-bit floating point (FP16 or BF16) down to 8-bit integers (INT8) or 4-bit integers (INT4) — to shrink memory footprint and accelerate inference. If you’ve wondered what is quantization LLM engineers rely on to fit 70B-parameter models on a single GPU, the answer is straightforward: it trades a small, measurable amount of model quality for dramatic reductions in VRAM and compute cost. This post covers the mechanics, the trade-offs, and the practical decisions you’ll face when quantizing models for production.

How quantization works

At its core, quantization maps a continuous range of high-precision values onto a discrete set of lower-precision values. The mapping is defined by a scale factor and a zero point:

quantized_value = round((float_value / scale) + zero_point)
dequantized_value = (quantized_value - zero_point) * scale

For symmetric quantization (common in LLM inference), the zero point is zero, simplifying to:

scale = max(abs(float_values)) / (2^(bits-1) - 1)
quantized = round(float_value / scale)

A BF16 weight matrix of shape [4096, 4096] occupies 32 MB. The same matrix in INT4 occupies 8 MB — a 4× reduction. Activations can be quantized similarly, though they’re often kept in higher precision (FP16 or INT8) during computation to avoid accuracy degradation.

Per-tensor vs. per-channel vs. group quantization

• Per-tensor: One scale for the entire tensor. Simple, but coarse — outliers in one channel degrade precision everywhere.
• Per-channel: One scale per output channel (for weights) or per token (for activations). Better accuracy, standard for INT8 weight-only quantization.
• Group quantization: Scales shared across small groups of weights (typically 32, 64, or 128 elements). The sweet spot for 4-bit: fine-grained enough to handle outliers, coarse enough to keep metadata overhead low. GPTQ, AWQ, and GGUF all use group quantization.

# Conceptual group quantization (pseudo-code)
def quantize_grouped(weight: torch.Tensor, group_size: int = 128, bits: int = 4):
    # weight shape: [out_features, in_features]
    orig_shape = weight.shape
    weight = weight.reshape(-1, group_size)
    
    # Per-group scale
    max_val = weight.abs().max(dim=1, keepdim=True).values
    scale = max_val / (2**(bits-1) - 1)
    
    # Quantize
    qweight = torch.round(weight / scale).clamp(-(2**(bits-1)), 2**(bits-1)-1)
    return qweight.to(torch.int8), scale, orig_shape

Why it matters for GPU inference

GPU memory bandwidth, not compute, is the bottleneck for LLM inference at typical batch sizes. Quantization attacks both dimensions:

The KV cache stays in FP16/INT8 regardless of weight quantization, so context length scaling is unaffected. But weight loading — the dominant cost for prefill and the limiting factor for model size — drops proportionally.

Real-world implication

A 70B model in FP16 needs ~140 GB VRAM (2× H100 80GB or 4× A100 40GB). The same model at INT4 fits on a single H100 80GB with room for KV cache and batch overhead. That’s the difference between “requires a cluster” and “runs on one GPU.”

Concrete example: Quantizing Llama-3-8B with AWQ

Activation-aware Weight Quantization (AWQ) observes that not all weights are equally important — it searches for per-channel scaling that protects salient weights (those multiplied by large activation magnitudes). The workflow:

# Install autoawq
pip install autoawq

# Quantize
python -m awq.quantize \
  --model_path meta-llama/Meta-Llama-3-8B \
  --quant_path ./Llama-3-8B-AWQ \
  --w_bit 4 \
  --q_group_size 128 \
  --zero_point \
  --version GEMM \
  --calib_data wikitext2 \
  --n_samples 128 \
  --seqlen 2048

Key flags explained:

• --w_bit 4: Target 4-bit weights
• --q_group_size 128: Group size (128 is standard for AWQ)
• --zero_point: Use asymmetric quantization (better for 4-bit)
• --version GEMM: Kernel layout optimized for GEMM (vs. GEMV for small batch)
• --calib_data: Calibration dataset — wikitext2 or c4 works; 128 samples of 2048 tokens is sufficient

The output directory contains qmodel.pt (quantized weights), quant_config.json, and the original tokenizer. Load it with:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model = AutoAWQForCausalLM.from_quantized("./Llama-3-8B-AWQ", fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained("./Llama-3-8B-AWQ")

# Inference
tokens = tokenizer("Quantization reduces memory by", return_tensors="pt").to("cuda")
output = model.generate(**tokens, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))

fuse_layers=True fuses QKV projections and applies other kernel fusions — critical for throughput. Without it, you’ll see 30–50% slower token generation.

Benchmarking the result

import torch
import time

# Warmup
for _ in range(3):
    model.generate(**tokens, max_new_tokens=16)

# Measure
torch.cuda.synchronize()
start = time.time()
output = model.generate(**tokens, max_new_tokens=256)
torch.cuda.synchronize()
elapsed = time.time() - start

num_tokens = output.shape[1] - tokens.input_ids.shape[1]
print(f"Generated {num_tokens} tokens in {elapsed:.2f}s = {num_tokens/elapsed:.1f} tok/s")
print(f"VRAM allocated: {torch.cuda.max_memory_allocated() / 1e9:.1f} GB")

On an A100 40GB, expect ~3.8 GB VRAM for the quantized model + KV cache at 4k context, and 80–120 tok/s single-stream depending on kernel optimization.

Common misconceptions

“Quantization always degrades quality significantly”

False for 4-bit weight-only quantization on modern architectures. AWQ and GPTQ typically recover 98–99% of FP16 performance on MMLU, GSM8K, and HumanEval for models ≥7B. The degradation is measurable on perplexity (1–2% relative increase) but often invisible on downstream tasks. Below 4-bit (3-bit, 2-bit), quality drops sharply — avoid unless you have no choice.

“INT8 quantization is free quality-wise”

INT8 weight-only is nearly free. INT8 weight + activation (W8A8) introduces activation quantization error that compounds across layers. For 70B+ models, W8A8 is often fine. For 7B–13B models, you may see 2–5% accuracy drops on reasoning benchmarks. Test your workload.

“You can quantize any model post-training”

Post-training quantization (PTQ) works well for dense LLMs. It fails or degrades severely on:

• Mixture-of-experts (MoE) models with expert routing sensitive to weight perturbation
• Models trained with unusual activation functions (e.g., SwiGLU variants with large dynamic ranges)
• Quantization-aware training (QAT) checkpoints — they’re already quantized; don’t re-quantize

For MoE, use AWQ with per-expert calibration or switch to QAT.

“GGUF and GPTQ are interchangeable”

They target different runtimes:

• GPTQ/AWQ/EXL2: GPU-first, require CUDA kernels, loaded via autoawq, auto-gptq, or exllamav2. Best throughput on NVIDIA GPUs.
• GGUF: CPU-first, runs on llama.cpp, supports metal/ROCm/CUDA backends. Best for Apple Silicon, consumer GPUs with limited VRAM, or hybrid CPU/GPU offload.

Don’t load a GGUF file in a GPTQ loader or vice versa. The quantization schemes differ (GGUF uses k-quant with mixed block sizes; GPTQ/AWQ use uniform group quantization).

“Smaller quantization = proportionally faster”

Throughput scales with memory bandwidth reduction only if you’re memory-bound. At large batch sizes or with small models, you become compute-bound. INT4 GEMM kernels also have lower occupancy than FP16 on current GPUs (Hopper improves this). Measure end-to-end latency at your target batch size — don’t assume 4× memory reduction means 4× speedup.

Choosing a quantization method

Integration note

If you’re serving multiple quantized models behind a single endpoint, the inference gateway needs to handle model-specific loading paths — AWQ models load via AutoAWQForCausalLM, GGUF via llama.cpp bindings, EXL2 via exllamav2. Routing requests to the right loader and tracking per-model VRAM usage is infrastructure work, not model work. An OpenAI-compatible gateway that abstracts this lets you swap quantization formats without changing client code.

Summary

Quantization is the highest-leverage optimization for LLM inference: 4× memory reduction with <2% quality loss on modern 4-bit methods. AWQ and GPTQ are the production standards for NVIDIA GPUs; GGUF dominates CPU and Apple Silicon. The mechanics are simple — group-wise affine mapping — but the implementation details (calibration data, group size, kernel fusion) determine whether you get a usable model or a degraded one. Quantize once, benchmark your actual workload, and treat the quantized artifact as a first-class model variant in your deployment pipeline.