What is 4-bit quantization and why does it matter?

4-bit quantization reduces each model weight from 16-bit floating point to a 4-bit integer representation, shrinking memory footprint by roughly 4x while preserving most model capability. The technique maps continuous weight values into 16 discrete buckets per quantization group, typically using per-channel or per-group scaling factors to minimize quantization error. For engineers deploying LLMs on constrained hardware, this is the single most impactful compression technique available today.

How 4-bit quantization works

Standard LLM weights live in FP16 or BF16 — 16 bits per parameter. A 7B parameter model at FP16 consumes about 14 GB of VRAM just for weights. 4-bit quantization packs four weights into a single 16-bit word, plus a small overhead for scaling factors.

The core operation is affine quantization:

quantized = round((weight - zero_point) / scale)
dequantized = quantized * scale + zero_point

Where scale and zero_point are computed per quantization group. A group might be 64 or 128 weights along the output channel dimension. This granularity matters: per-tensor scaling (one scale for the whole matrix) destroys accuracy; per-group scaling preserves it.

Two dominant formats exist today:

NF4 (NormalFloat 4-bit) — Used by bitsandbytes and QLoRA. The 16 quantization levels are spaced according to a normal distribution, matching the empirical distribution of pretrained weights. This allocates more precision near zero where weights concentrate.

INT4 symmetric — Simpler, used by GPTQ, AWQ, and most inference engines. Levels are evenly spaced: {-8, -7, ..., 7} with an implicit zero point at 0. Scale is max(abs(weight)) / 7.

Both store weights as packed uint8 arrays. Dequantization happens on the fly during matrix multiplication, either in custom kernels (CUDA, Metal, Triton) or via tensor cores with native INT4 support on Hopper and Blackwell GPUs.

# Conceptual NF4 dequantization (bitsandbytes approach)
NF4_VALUES = torch.tensor([
    -1.0, -0.696, -0.525, -0.395, -0.284, -0.185, -0.091, 0.0,
    0.079, 0.161, 0.246, 0.338, 0.441, 0.562, 0.720, 1.0
])

def dequantize_nf4(packed_uint8, scale, block_size=64):
    # Unpack 2 weights per byte
    indices = torch.stack([
        packed_uint8 & 0xF,
        (packed_uint8 >> 4) & 0xF
    ], dim=-1).view(-1)
    return NF4_VALUES[indices].view(-1, block_size) * scale.view(-1, 1)

Why it matters for deployment

Memory bandwidth, not compute, bounds LLM inference on modern hardware. A 70B model at FP16 needs 140 GB VRAM — four H100s or eight A100s. At 4-bit, it fits on two H100s (80 GB each) or a single H200 (141 GB). This changes the economics entirely.

The bandwidth math: generating one token requires reading all weights once. At 4-bit, you move 4x less data from VRAM to SMs. On memory-bound workloads (batch size < 32, typical for serving), this translates directly to 2-3x higher throughput per GPU.

But the real win is enabling models that otherwise wouldn’t fit. A 32B model at 4-bit runs on a 24 GB consumer GPU (RTX 3090/4090). The same model at FP16 needs 64 GB — datacenter hardware only. This democratization is why 4-bit became the default for local inference.

Quantization also reduces KV cache pressure indirectly: smaller activations from quantized linear layers mean less intermediate memory, though KV cache itself remains FP16/BF16 in most implementations.

Concrete example: quantizing Llama-3-8B

Start with the FP16 checkpoint. Apply GPTQ with 128-group size, act-order=True, dampening=0.01. Calibration uses 128 sequences of 2048 tokens from the training distribution (or a representative sample like WikiText-2).

# Using auto-gptq
python quantize.py \
  --model meta-llama/Meta-Llama-3-8B \
  --bits 4 \
  --group-size 128 \
  --act-order \
  --dataset wikitext2 \
  --nsamples 128 \
  --output-dir llama-3-8b-gptq-4bit

Result: 8B parameters → ~4.7 GB (vs 16 GB FP16). Perplexity on WikiText-2 increases from 5.12 to ~5.35 — a 4.5% relative degradation. MMLU drops 1-2 points absolute. Human eval shows negligible difference for most tasks.

The quantized model loads in transformers with:

from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

model = AutoGPTQForCausalLM.from_quantized(
    "llama-3-8b-gptq-4bit",
    device_map="auto",
    use_triton=True,  # faster kernels on Ampere+
    inject_fused_attention=True,
    inject_fused_mlp=True,
)
tokenizer = AutoTokenizer.from_pretrained("llama-3-8b-gptq-4bit")

On an H100, this serves ~4,500 tokens/sec at batch=1 (vs ~1,800 for FP16). On a 4090, ~1,200 tokens/sec — fast enough for real-time chat.

Common misconceptions

“4-Bit always means 4-bit everything”

Most “4-bit” models quantize only linear weights. Embeddings, output head, normalization parameters (RMSNorm/LayerNorm), and KV cache stay FP16 or BF16. A true 4-bit model would quantize everything — but norm statistics and embeddings are extremely sensitive. The 4.7 GB figure for Llama-3-8B includes FP16 embeddings (8M vocab × 4096 dim × 2 bytes ≈ 64 MB) and norms. Actual weight savings is closer to 3.8x, not 4x.

“QLoRA fine-tunes in 4-bit”

QLoRA freezes the 4-bit base model and trains LoRA adapters in FP16/BF16. The base weights never update. Gradients flow through dequantized weights during backward pass, but the 4-bit values stay fixed. This works because LoRA adapters absorb task-specific changes. Full fine-tuning in 4-bit (updating quantized weights) remains an open research problem — gradient quantization noise accumulates catastrophically.

“Lower bits always degrade quality linearly”

The accuracy curve is not linear. 8-bit → 4-bit costs ~1-2% perplexity. 4-bit → 3-bit costs ~5-8%. 3-bit → 2-bit destroys most models. The cliff appears around 3.5 bits for dense models. Mixture-of-experts models tolerate lower bits better because expert specialization creates sparser weight distributions.

“Group size doesn’t matter much”

Group size is the single most important hyperparameter after bit width. Group=128 is the practical minimum for 4-bit. Group=32 recovers ~50% of the FP16-4-bit gap but increases metadata overhead (more scales). Group=64 is a common sweet spot. Group=1 (per-channel) is ideal but rarely used because scale storage approaches weight storage.

# Metadata overhead at different group sizes (Llama-3-8B, 4-bit)
# Weights: 8B * 4 bits = 4 GB
# Scales (FP16): 8B / group_size * 2 bytes
group_size=128:  128 MB scales  (3.2% overhead)
group_size=64:   256 MB scales  (6.4% overhead)
group_size=32:   512 MB scales  (12.8% overhead)
group_size=1:    16 GB scales   (400% overhead — defeats the purpose)

“All 4-bit formats are interchangeable”

NF4 and INT4 are not drop-in replacements. NF4’s non-uniform spacing matches pretrained weight distributions better, giving ~0.1-0.2 perplexity improvement over INT4 at same group size. But NF4 requires custom dequantization kernels; INT4 maps to native tensor core instructions on H100 (via ldmatrix + mma.sync with .b4 type). AWQ uses INT4 with activation-aware scaling — it searches per-channel scales that minimize activation quantization error, not weight error. This matters for models with outlier channels.

“Quantization-aware training (QAT) solves everything”

QAT simulates quantization during training with straight-through estimators. It recovers most of the accuracy gap but requires full training compute — you’re retraining the model. For open-weight models, someone else did the QAT (e.g., Nemotron-3-8B-4B-QAT). For proprietary models, you only have post-training quantization (PTQ). PTQ at 4-bit with good calibration data is 95% of QAT for 1/1000th the cost.

Calibration data quality matters more than algorithm

GPTQ, AWQ, RTN (round-to-nearest), and SpQR all produce similar results if calibration data matches the deployment distribution. The algorithm choice matters less than using 128-256 sequences from your actual task domain.

# Bad: random tokens
calib_data = torch.randint(0, vocab_size, (128, 2048))

# Good: representative text
from datasets import load_dataset
ds = load_dataset("your-domain-corpus", split="train[:1000]")
calib_data = tokenizer(ds["text"], max_length=2048, truncation=True, padding="max_length")

Domain mismatch (e.g., calibrating on Wikipedia for a code model) costs 2-3x more perplexity than the choice between GPTQ and AWQ.

When not to use 4-bit

• High-precision reasoning tasks: Math, code generation with strict syntax, multi-step logic. The 1-2% perplexity increase compounds across reasoning steps.
• Embedding models: Retrieval quality degrades sharply. Use 8-bit or FP16 for embeddings; quantize only the generator in RAG pipelines.
• Training from scratch: Quantization-aware training exists but adds complexity. Train in BF16, quantize for deployment.
• Models with extreme outliers: Some MoE routers or attention heads have weight distributions that 4-bit cannot represent without massive error. AWQ helps; sometimes you need mixed precision (4-bit for most layers, 8-bit for sensitive ones).

The current state of tooling

GGUF’s k-quant variants (Q4_K_M, Q4_K_S) mix 4-bit and 8-bit blocks based on weight importance — a practical hack that outperforms uniform 4-bit on CPU where dequantization overhead dominates.

What’s next

FP8 (E4M3/E5M2) on Hopper/Blackwell offers 2x compression over FP16 with near-lossless accuracy — no calibration, no PTQ artifacts. But FP8 requires H100/H200/B200. 4-bit remains the only option for Ampere, consumer GPUs, Apple Silicon, and CPUs.

Research pushes toward 3-bit with outlier preservation (SpQR, PB-LLM, QuIP#) and 2-bit with learned codebooks (PVQ, VQ-VAE). None are production-ready for general models yet.

For now: 4-bit PTQ with group_size=128, calibration from your domain, AWQ or GPTQ depending on your serving stack. It works, it’s stable, and it fits on the hardware you have.