How model quantization reduces memory and cost

Quantization is the single most effective lever for running large language models on commodity hardware. By reducing weight precision from 16-bit floats to 4-bit integers, you can shrink a 70B parameter model from 140 GB to roughly 40 GB — putting it within reach of a single 48 GB GPU or even high-end consumer hardware. This guide walks through how quantization reduces model size, the three dominant formats you’ll encounter in production, and how to verify your quantized model hasn’t regressed beyond your quality threshold.

Step 1: Choose your quantization format

Three formats dominate the open-weight ecosystem. Pick one based on your deployment target:

Rule of thumb: If you’re targeting consumer GPUs or CPU-only machines, use GGUF. If you’re serving on NVIDIA GPUs with vLLM or TGI, prefer AWQ for speed or GPTQ for broader compatibility. AWQ typically edges out GPTQ on latency at 4-bit because it protects salient weights during quantization without requiring per-channel scaling at runtime.

Step 2: Set up a quantization environment

You need a machine with enough VRAM to hold the unquantized model temporarily — typically 2× the target quantized size. For a 70B model at BF16, that’s ~140 GB VRAM (8× H100 80GB or 4× A100 80GB). If you lack that, quantize a smaller model first (7B–13B) to validate your pipeline.

# Create isolated environment
python -m venv quant-env && source quant-env/bin/activate

# Core dependencies (adjust CUDA version as needed)
pip install --upgrade pip
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install autoawq auto-gptq transformers accelerate datasets huggingface_hub

For GGUF quantization, you’ll also need llama.cpp compiled with your target backend:

# CPU-only (fastest to get started)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make -j$(nproc)

# Or with CUDA support
make LLAMA_CUBLAS=1 -j$(nproc)

Step 3: Quantize with AWQ (recommended for NVIDIA GPU serving)

AWQ (Activation-aware Weight Quantization) identifies the 1% of weights most sensitive to quantization error using a small calibration set, then quantizes only the remaining 99% to 4-bit. This preserves quality better than naive PTQ.

# quantize_awq.py
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
import torch

model_id = "meta-llama/Meta-Llama-3-70B-Instruct"
quant_path = "./Llama-3-70B-Instruct-AWQ-4bit"

# Load model in BF16 (requires ~140 GB VRAM for 70B)
model = AutoAWQForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Calibration data: 512 samples of 512 tokens each from your target domain
# For general chat, use a slice of UltraChat or ShareGPT
from datasets import load_dataset
calib_data = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:512]")
calib_texts = [tokenizer.apply_chat_template(
    msg, tokenize=False, add_generation_prompt=False
) for msg in calib_data["messages"]]

# Quantize: w_bit=4, q_group_size=128, zero_point=True (asymmetric)
model.quantize(
    tokenizer,
    calib_data=calib_texts,
    w_bit=4,
    q_group_size=128,
    zero_point=True,
    version="GEMM",  # "GEMV" for batch-size-1 latency optimization
)

# Save quantized model and tokenizer
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
print(f"Saved AWQ model to {quant_path}")

Run it:

python quantize_awq.py

Expected output: a directory containing model.safetensors (quantized weights), quant_config.json, and tokenizer files. The safetensors file should be ~40 GB for 70B at 4-bit.

Step 4: Quantize with GPTQ (alternative for vLLM/TGI compatibility)

GPTQ uses layer-wise reconstruction with Hessian information. It’s slower to quantize but widely supported.

# quantize_gptq.py
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer
import torch

model_id = "meta-llama/Meta-Llama-3-70B-Instruct"
quant_path = "./Llama-3-70B-Instruct-GPTQ-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False,  # True = act-order, slower inference, slightly better quality
    sym=True,        # symmetric quantization
    true_sequential=True,
)

model = AutoGPTQForCausalLM.from_pretrained(
    model_id,
    quantize_config,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    device_map="auto",
)

# Calibration dataset (same pattern as AWQ)
from datasets import load_dataset
calib_data = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:512]")
calib_texts = [tokenizer.apply_chat_template(
    msg, tokenize=False, add_generation_prompt=False
) for msg in calib_data["messages"]]

model.quantize(calib_texts, batch_size=1, use_triton=True)
model.save_quantized(quant_path, use_safetensors=True)
tokenizer.save_pretrained(quant_path)

Key difference: desc_act=True (act-order) improves perplexity by ~0.1-0.2 but adds a permutation step at inference that hurts throughput. For serving, desc_act=False is usually the right call.

Step 5: Quantize with GGUF (for CPU, Apple Silicon, or hybrid offload)

GGUF quantization happens after model conversion. You first convert the HF model to GGUF (FP16), then quantize down. This two-step process lets you produce multiple quantization levels from one FP16 GGUF.

# Step 5a: Convert HF -> GGUF (FP16)
cd llama.cpp
python convert_hf_to_gguf.py \
  /path/to/Meta-Llama-3-70B-Instruct \
  --outfile Llama-3-70B-Instruct-f16.gguf \
  --outtype f16

# Step 5b: Quantize to 4-bit (Q4_K_M = recommended default)
./llama-quantize \
  Llama-3-70B-Instruct-f16.gguf \
  Llama-3-70B-Instruct-Q4_K_M.gguf \
  Q4_K_M

Common GGUF quantization levels (descending quality/size):

The _K suffix means “k-quant” — mixed precision per layer (attention vs. FFN). _M means medium (balanced). Avoid legacy q4_0, q4_1 etc.; they’re strictly worse than k-quants.

Step 6: Verify quantization quality (automated)

Never ship a quantized model without automated evaluation. Perplexity on a held-out set is the fastest signal; task-specific evals are the ground truth.

# eval_perplexity.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
from tqdm import tqdm
import math

def evaluate_ppl(model, tokenizer, dataset_name="wikitext", split="test", stride=512, max_samples=100):
    data = load_dataset(dataset_name, "wikitext-2-raw-v1", split=split)
    texts = "\n\n".join(data["text"][:max_samples])
    encodings = tokenizer(texts, return_tensors="pt", truncation=False)
    input_ids = encodings.input_ids.to(model.device)
    
    nlls = []
    seq_len = input_ids.size(1)
    
    for i in tqdm(range(0, seq_len, stride)):
        end_loc = min(i + stride, seq_len)
        trg_len = end_loc - i
        input_batch = input_ids[:, i:end_loc]
        target_batch = input_batch.clone()
        target_batch[:, :-trg_len] = -100
        
        with torch.no_grad():
            outputs = model(input_batch, labels=target_batch)
            nlls.append(outputs.loss * trg_len)
    
    ppl = math.exp(torch.stack(nlls).sum() / (seq_len - stride))
    return ppl

# Load quantized model (example: AWQ)
model = AutoModelForCausalLM.from_pretrained(
    "./Llama-3-70B-Instruct-AWQ-4bit",
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("./Llama-3-70B-Instruct-AWQ-4bit")

ppl = evaluate_ppl(model, tokenizer)
print(f"Wikitext-2 PPL: {ppl:.2f}")

Baseline targets for Llama-3-70B-Instruct (approximate, wikitext-2):

• FP16/BF16: ~2.85
• AWQ 4-bit / GPTQ 4-bit: ~2.95–3.05
• GGUF Q4_K_M: ~3.00–3.10
• GGUF Q3_K_M: ~3.30–3.50

If your perplexity exceeds the baseline by >0.3, something went wrong — usually calibration data mismatch or wrong group size.

Step 7: Verify quantization quality (task-specific)

Perplexity correlates with downstream performance but doesn’t guarantee it. Run at least one task eval relevant to your use case.

# eval_mmlu.py (requires lm-evaluation-harness)
# pip install lm-eval
import subprocess
import json

def run_mmlu(model_path, model_type="hf", batch_size=4):
    cmd = [
        "lm_eval",
        "--model", model_type,
        "--model_args", f"pretrained={model_path},dtype=auto",
        "--tasks", "mmlu",
        "--batch_size", str(batch_size),
        "--output_path", "./eval_results",
        "--log_samples",
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    print(result.stdout)
    return result.returncode == 0

# For GGUF models, use llama.cpp backend
def run_mmlu_gguf(gguf_path, batch_size=4):
    cmd = [
        "lm_eval",
        "--model", "gguf",
        "--model_args", f"model_path={gguf_path},n_gpu_layers=-1",
        "--tasks", "mmlu",
        "--batch_size", str(batch_size),
        "--output_path", "./eval_results_gguf",
    ]
    subprocess.run(cmd, check=True)

Target: MMLU 5-shot within 1-2% of FP16 baseline. For Llama-3-70B-Instruct FP16 ~82%, expect AWQ/GPTQ 4-bit ~80-81%, GGUF Q4_K_M ~79-80%.

Step 8: Deploy and measure serving metrics

Quantization reduces model size, but the serving metrics that matter are throughput (tokens/sec), latency (TTFT, TPOT), and VRAM footprint. Load your quantized model in your serving stack and benchmark.

VLLM (AWQ/GPTQ)

# serve_vllm.py
from vllm import LLM, SamplingParams

llm = LLM(
    model="./Llama-3-70B-Instruct-AWQ-4bit",
    quantization="awq",  # or "gptq"
    tensor_parallel_size=2,  # adjust for GPU count
    gpu_memory_utilization=0.9,
    max_model_len=8192,
    trust_remote_code=True,
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

prompts = [
    "Explain how quantization reduces model size in three paragraphs.",
    "Write a Python function that computes the Fibonacci sequence iteratively.",
] * 10  # 20 requests

outputs = llm.generate(prompts, sampling_params)
for out in outputs:
    print(out.outputs[0].text[:100])

Key vLLM flags for quantized models:

• quantization="awq" or "gptq" — enables fused kernels
• max_num_seqs — tune for your batch latency budget
• enforce_eager=True — disable CUDA graph capture if you hit OOM (graphs add ~10-15% VRAM)

Llama.cpp server (GGUF)

# CPU-only, 8 threads
./llama-server -m Llama-3-70B-Instruct-Q4_K_M.gguf -c 8192 -t 8 --port 8080

# GPU offload (Metal on macOS, CUDA on Linux)
./llama-server -m Llama-3-70B-Instruct-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080

-ngl 99 offloads all possible layers to GPU. On a 24 GB VRAM card, a 70B Q4_K_M (~40 GB) will split ~60/40 GPU/CPU. Monitor with htop and nvidia-smi / sudo powermetrics to verify offload ratio.

Step 9: Profile memory and latency

# profile_serving.py
import time
import torch
from vllm import LLM, SamplingParams

llm = LLM(
    model="./Llama-3-70B-Instruct-AWQ-4bit",
    quantization="awq",
    tensor_parallel_size=2,
    gpu_memory_utilization=0.9,
    max_model_len=4096,
)

# Warmup
_ = llm.generate(["warmup"], SamplingParams(max_tokens=10))

# Measure TTFT (time to first token) and throughput
prompts = ["Write a detailed explanation of quantum computing."] * 8
sampling = SamplingParams(max_tokens=256, temperature=0.7)

start = time.perf_counter()
outputs = llm.generate(prompts, sampling)
end = time.perf_counter()

total_tokens = sum(len(o.outputs[0].token_ids) for o in outputs)
elapsed = end - start
throughput = total_tokens / elapsed

# TTFT approximation: time until first token across all requests
# vLLM doesn't expose per-request TTFT directly; use streaming for precise measurement
print(f"Total tokens: {total_tokens}")
print(f"Elapsed: {elapsed:.2f}s")
print(f"Throughput: {throughput:.1f} tok/s")
print(f"Per-request latency: {elapsed/len(prompts)*1000:.0f} ms")

Expected ballparks for 70B Q4 on 2× H100 (80 GB):

• Throughput: 3,000–5,000 tok/s (batch 8, 256 output tokens)
• TTFT: 50–150 ms (depends on queue depth)
• VRAM: ~45 GB total (model + KV cache + workspace)

If VRAM exceeds gpu_memory_utilization, reduce max_model_len or max_num_seqs.

Step 10: Automate regression testing in CI

Quantization is a lossy transform. Treat quantized models as artifacts that need versioning and regression gates.

# .github/workflows/quantization-check.yml
name: Quantization Quality Gate
on:
  workflow_dispatch:
  push:
    tags: ['quant-*']

jobs:
  eval:
    runs-on: [self-hosted, gpu, a100-80gb]
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - name: Install deps
        run: |
          pip install -r requirements-quant.txt
      - name: Download quantized model
        run: |
          huggingface-cli download my-org/Llama-3-70B-Instruct-AWQ-4bit --local-dir ./model
      - name: Run perplexity check
        run: |
          python eval_perplexity.py --model ./model --threshold 3.1
      - name: Run MMLU eval
        run: |
          python eval_mmlu.py --model ./model --threshold 0.79
      - name: Publish results
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: ./eval_results/

The thresholds (--threshold 3.1 for PPL, --threshold 0.79 for MMLU accuracy) should be set from your baseline FP16 runs minus your acceptable regression budget.

Common failure modes and fixes

When to stop quantizing

4-bit (AWQ/GPTQ/GGUF Q4_K_M) is the production sweet spot for most teams. Pushing to 3-bit saves ~20% more memory but typically costs 3-5% MMLU and noticeable reasoning degradation. 2-bit is a research artifact — don’t serve it.

If 4-bit still doesn’t fit your hardware, the correct engineering move is not 3-bit quantization. It’s:

1. Model distillation (train a smaller student)
2. Tensor parallelism across more GPUs
3. CPU offload with GGUF (accept latency penalty)
4. A smaller base model (Llama-3-8B vs 70B)

Quantization reduces model size by compressing weights, but it doesn’t change the fundamental compute graph. Know the difference.