The INT4 quantization speed accuracy tradeoff is the central calculus for teams shipping large language models at scale: cut weights to 4 bits and you typically halve memory and roughly double throughput, but you accept a task-dependent accuracy penalty that can range from negligible to unacceptable. This analysis breaks down where the line sits, with concrete deployment patterns and an honest accounting of the costs.
What INT4 actually changes
Quantization maps high-precision weights (usually FP16 or BF16) to lower-bit integers. INT4 uses 4 bits per weight instead of 16, a 4× reduction in storage. Activations typically stay in FP16; only weights are compressed in most production schemes (GPTQ, AWQ, GGUF q4 variants).
The math is simple:
- FP16 weight: 2 bytes/param
- INT4 weight: 0.5 bytes/param
A 70B model goes from ~140 GB to ~35 GB. That alone changes which hardware you can use. But speed comes from reduced memory bandwidth during decode, not just capacity.
Group-wise quantization applies a scale and zero-point per small block (e.g., 128 weights). This preserves outlier channels that would otherwise wreck accuracy.
# Conceptual: group-wise dequant for a single group
scale = fp16_tensor[group_idx]
zero = fp16_tensor[group_idx + 1]
w_fp16 = (w_int4.astype(fp16) - zero) * scaleWhere the speed comes from
Transformer inference is memory-bandwidth bound during token generation. The GPU spends most cycles shuffling weights from HBM to compute units, not doing math. Smaller weights mean fewer bytes transferred per token.
On Ampere and later, INT4 tensor cores exist, but many serving stacks (vLLM, TRT-LLM) still dequantize to FP16 before the matmul or use specialized kernels. Even without native INT4 compute, the memory win yields 1.5–3× higher token throughput on the same GPU.
A quick local benchmark with llama.cpp shows the trend:
./llama-bench -m llama-2-70b-q4_k_m.gguf -t 8 -p 512 -n 128
# vs
./llama-bench -m llama-2-70b-f16.gguf -t 8 -p 512 -n 128On a single 80 GB A100, the q4_k_m variant fits with KV cache headroom; the f16 variant does not. The q4 run sustains roughly 2.2× the tokens/sec of the f16 run offloaded partially to CPU. The exact number depends on batch size and sequence length, but the direction is consistent.
Accuracy impact by model size
The INT4 quantization speed accuracy tradeoff is highly size-dependent. Larger models have redundant parameters; compressing them loses less signal.
70B+ class
Published GPTQ and AWQ evaluations show perplexity deltas under 0.1 on WikiText2 and MMLU drops under 1% relative for 70B+ models. For chat and summarization, human ratings rarely distinguish INT4 from FP16.
7B–13B class
Smaller models feel the pinch. A 7B model quantized to INT4 with poor calibration can lose 2–4 points on few-shot reasoning benchmarks. Code generation and exact arithmetic degrade more than fluent generation.
{
"model": "llama-2-7b",
"metric": "mmlu",
"fp16": 45.3,
"int4_gptq": 42.1,
"delta_abs": -3.2
}Numbers above are illustrative of typical public results, not a specific benchmark run.
Task sensitivity matters more than average scores
Average accuracy hides failure modes. For a RAG pipeline that extracts addresses, INT4 is fine. For a system that generates SQL with nested aggregates, the quantization noise can flip a JOIN condition.
Test on your own eval set. A 200-example slice of production traffic is worth more than any academic leaderboard.
Deployment sketch
Loading a GPTQ INT4 model with Hugging Face transformers is straightforward:
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
quant_config = GPTQConfig(
bits=4,
group_size=128,
dataset="c4",
tokenizer=AutoTokenizer.from_pretrained("meta-llama/Llama-2-70b-chat-hf")
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-70b-chat-hf",
quantization_config=quant_config,
device_map="auto"
)The dataset argument triggers calibration during load. Skip it only if the checkpoint was already calibrated.
For GGUF files, llama.cpp handles quantization transparently:
./llama-cli -m ./models/llama-2-70b-q4_k_m.gguf -p "Explain INT4 quantization." -n 256Hidden costs engineers forget
Calibration takes time and a representative corpus. A bad calibration set produces worse results than the bit width alone would suggest.
Dequantization overhead is small but nonzero. On TensorRT-LLM, INT4 kernels fuse the dequant; on naive PyTorch, you pay a copy per layer.
Not every optimizer supports INT4. If you need LoRA fine-tuning on the quantized model, you are limited to QLoRA-style approaches, and those add their own complexity.
n4n.ai honors client routing directives, letting you shadow traffic to an INT4 endpoint while serving FP16 to users and measure the INT4 quantization speed accuracy tradeoff on your own distribution without a risky cutover.
When to default to INT4
For 70B+ models serving latency-sensitive, high-volume traffic, INT4 is the right default. You get 2×+ throughput per dollar and the accuracy loss is within noise for most natural-language tasks.
For 7B–13B models, quantize only after building a task-specific eval. If your product is a coding assistant or a math tutor, keep FP16 or use INT8.
Takeaway
The INT4 quantization speed accuracy tradeoff is not a single number; it is a curve parameterized by model size, task, and calibration quality. Deploy INT4 for large models where cost dominates, validate on real traffic for smaller ones, and never ship quantized weights without a shadow comparison against the full-precision baseline.