Quantization is the single most effective lever for running large language models on commodity hardware, but every engineer who has tried it has asked the same question: does quantization reduce accuracy? The short answer is yes, but the magnitude depends entirely on the method, the model scale, and the task. Understanding where the degradation lives — and where it doesn’t — lets you make deliberate tradeoffs instead of guessing.
The precision ladder
Before evaluating accuracy impact, you need to distinguish between quantization schemes. They are not created equal.
Post-training quantization (PTQ) takes a trained FP16 or BF16 model and compresses weights after the fact. No retraining, no data required beyond a small calibration set. Common variants:
- INT8: 8-bit integer weights, typically per-tensor or per-channel scaling. Negligible accuracy loss on most models 7B and above.
- INT4: 4-bit integer weights. The workhorse for consumer GPUs. Quality drops noticeably on smaller models (<7B) and reasoning-heavy tasks.
- GPTQ / AWQ / EXL2: PTQ methods that use layer-wise reconstruction or activation-aware scaling to recover INT4 quality. AWQ is currently the best default for LLaMA-family models.
Quantization-aware training (QAT) inserts fake-quantization nodes during training so the model learns to be robust to low precision. This recovers most INT4 accuracy but requires full training compute — rarely practical for open-weight models.
FP8 / BF8 (E4M3 / E5M2) are 8-bit floating-point formats supported natively on H100 and newer. They behave like FP16 with half the bandwidth. Accuracy is effectively indistinguishable from FP16 for inference.
# Typical PTQ workflow with AutoAWQ
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = "meta-llama/Llama-3-8B"
quant_path = "Llama-3-8B-AWQ"
model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
quant_config = {
"zero_point": True,
"q_group_size": 128,
"w_bit": 4,
"version": "GEMM"
}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)Where accuracy actually drops
The literature and production experience converge on three axes where quantization hurts.
1. Model scale
Smaller models have less redundancy. An INT4 7B model typically loses 5–15% relative performance on MMLU compared to its FP16 baseline. The same quantization on a 70B model often loses <2%. The parameter count acts as a buffer — overparameterized models tolerate aggressive compression because many weights are effectively noise.
# Rough relative MMLU degradation (FP16 = 100%)
# 7B INT4: 85-90%
# 13B INT4: 92-95%
# 70B INT4: 97-99%If you are running 7B or 3B models on edge devices, expect measurable degradation on complex reasoning, coding, and multilingual tasks. For 70B+ models, INT4 is often indistinguishable from FP16 in blind evals.
2. Task type
Quantization disproportionately affects tasks requiring:
- Multi-step reasoning (GSM8K, BBH): Accumulated rounding error compounds across steps.
- Code generation (HumanEval, MBPP): Syntax sensitivity amplifies small logit shifts.
- Long-context retrieval: Attention score precision matters more when comparing thousands of tokens.
Conversely, classification, summarization, and chat on short contexts often show zero measurable difference at INT4.
3. Calibration data quality
PTQ methods need a calibration set to compute scaling factors. Using 128 random Wikipedia passages works poorly for code models. Using 512 samples from your actual prompt distribution (or a close proxy) can recover 2–3% absolute accuracy on domain tasks.
# Better calibration: sample from your production prompt logs
def get_calibration_data(tokenizer, n_samples=512, max_len=2048):
# Replace with your actual prompt distribution
prompts = load_your_production_prompts(n_samples)
return tokenizer(
prompts,
padding=True,
truncation=True,
max_length=max_len,
return_tensors="pt"
).input_idsKV cache quantization: a separate decision
Weight quantization gets the attention, but KV cache quantization often matters more for throughput. The KV cache grows with sequence length and batch size — at 8K context and batch 32, a 70B model’s KV cache exceeds 20 GB in FP16.
Quantizing KV to INT8 (per-token or per-head) is nearly free on accuracy. INT4 KV starts to hurt retrieval and long-context coherence. FP8 KV on H100 is the sweet spot: half the memory, no measurable quality loss.
# vLLM KV cache quantization (FP8 on H100)
from vllm import EngineArgs
engine_args = EngineArgs(
model="meta-llama/Llama-3-70B",
kv_cache_dtype="fp8", # requires H100 + CUDA 12.4+
quantization="awq", # weight quantization
max_model_len=8192,
gpu_memory_utilization=0.90
)Measuring it yourself: the eval harness you need
Published benchmarks are necessary but insufficient. Your prompt distribution, your latency budget, and your quality bar are unique. Build a lightweight eval that runs nightly.
# Minimal eval harness structure
import json
from dataclasses import dataclass
from typing import List
from vllm import LLM, SamplingParams
@dataclass
class EvalCase:
prompt: str
expected_contains: List[str] # or use LLM-as-judge
max_tokens: int = 512
def evaluate(model_path: str, cases: List[EvalCase], quantization: str) -> dict:
llm = LLM(model=model_path, quantization=quantization, dtype="auto")
sampling = SamplingParams(temperature=0, max_tokens=512)
outputs = llm.generate([c.prompt for c in cases], sampling)
results = []
for case, output in zip(cases, outputs):
text = output.outputs[0].text
passed = all(needle in text for needle in case.expected_contains)
results.append({"prompt": case.prompt, "passed": passed, "output": text})
pass_rate = sum(r["passed"] for r in results) / len(results)
return {"pass_rate": pass_rate, "details": results}
# Example cases for a coding assistant
cases = [
EvalCase(
prompt="Write a Python function that computes the Levenshtein distance between two strings.",
expected_contains=["def levenshtein", "dynamic programming", "matrix"]
),
EvalCase(
prompt="Explain the difference between async/await and threading in Python.",
expected_contains=["GIL", "event loop", "blocking"]
),
]
# Run against FP16 baseline, then AWQ INT4, then GPTQ INT4
for quant in [None, "awq", "gptq"]:
print(f"Quantization: {quant or 'fp16'}")
print(evaluate("meta-llama/Llama-3-8B", cases, quant))Run this against your actual prompts. If INT4 passes your bar, ship it. If not, try INT8 or FP8 before assuming you need FP16.
The hardware reality check
Quantization is not purely an accuracy decision — it is a hardware utilization decision.
| GPU | VRAM | FP16 70B | INT4 70B | FP8 70B |
|---|---|---|---|---|
| A10G (24GB) | 24GB | ✗ | ✓ (offload) | ✗ |
| A100 (40GB) | 40GB | ✗ | ✓ | ✗ |
| A100 (80GB) | 80GB | ✓ (2×) | ✓ (4×) | ✓ (2×) |
| H100 (80GB) | 80GB | ✓ (2×) | ✓ (4×) | ✓ (2×, faster) |
On 24–40 GB GPUs, INT4 is the only way to run 70B models without tensor parallelism or offloading. The accuracy hit is the price of admission. On 80 GB+ GPUs, you have choices: FP8 gives you FP16 quality with INT4-like memory footprint, but only on Hopper.
Throughput matters too. INT4 GEMM kernels on Ampere (A100) are 2–2.5× faster than FP16. On Hopper, FP8 tensor cores are 2× faster than FP16 and match INT4 throughput with better accuracy. If you are latency-bound, the quantization scheme that fits in VRAM and uses the fastest kernel wins — even if it costs 1% accuracy.
When to use what: a decision framework
START: What is your model size?
├── ≤ 7B
│ ├── Consumer GPU (24GB or less) → AWQ INT4, accept degradation
│ ├── Datacenter GPU (40GB+) → FP8 if H100, else INT8
│ └── Quality-critical → QAT INT4 or stay FP16
├── 13B–34B
│ ├── Consumer GPU → AWQ INT4 (quality usually acceptable)
│ ├── Datacenter GPU → FP8 (H100) or INT8 (Ampere)
│ └── Quality-critical → INT8 or FP16
└── 70B+
├── 24–40GB GPU → AWQ INT4 (only option)
├── 80GB Ampere → INT4 for throughput, INT8 for quality
└── 80GB Hopper → FP8 (default), INT4 only if max throughput neededThe n4n.ai angle
We see this play out across thousands of deployments. Teams that treat quantization as a one-time checkbox (“just use 4-bit”) end up with silent quality regressions on edge cases. Teams that build eval harnesses, test against their actual prompt distribution, and treat quantization as a tunable parameter — alongside temperature, top-p, and context length — ship faster and iterate with confidence. The gateway layer should make it trivial to swap quantization schemes per model per request, not lock you into a single choice at deploy time.
The decisive takeaway
Quantization reduces accuracy, but the reduction is predictable and manageable. For models 13B and above, AWQ INT4 loses <3% relative on most benchmarks and is often indistinguishable in blind chat evals. For 70B models on H100, FP8 gives you FP16 quality at half the memory and 2× the throughput — there is no reason to run FP16. For 7B models on consumer GPUs, you will feel the degradation on reasoning and code; mitigate it with better calibration data, INT8 weights, or accept it as the cost of local inference.
Do not guess. Run your evals. The right quantization scheme is the one that passes your quality bar at the lowest cost per token.