FP16 vs INT8: how weight precision affects model size

Weight precision fp16 int8 decisions shape every downstream constraint: GPU memory, cold-start latency, batch throughput, and whether a model fits on your target hardware at all. The difference between 16-bit floats and 8-bit integers is not academic — it determines if a 7B model runs on a 24 GB VRAM card or requires two. This post breaks down the concrete trade-offs so you can pick without guessing.

How weight precision works

FP16 stores each parameter as a 16-bit IEEE 754 floating-point value: 1 sign bit, 5 exponent bits, 10 mantissa bits. Dynamic range spans roughly ±65,500 with ~3 decimal digits of precision. Most open-weight models (Llama, Mistral, Qwen) ship in FP16 or BF16 checkpoints because training in lower precision requires quantization-aware training (QAT) or careful loss scaling.

INT8 represents each weight as a signed 8-bit integer (−128 to 127). Since neural network weights cluster near zero, a linear mapping wastes dynamic range. Practical quantization uses per-tensor or per-channel scale factors:

# Per-tensor symmetric quantization (simplified)
scale = max(abs(w.max()), abs(w.min())) / 127
w_int8 = torch.round(w / scale).clamp(-128, 127).to(torch.int8)

# Dequantize for compute
w_fp16 = (w_int8 * scale).to(torch.float16)

Per-channel scaling (one scale per output channel) recovers ~0.5–1% accuracy over per-tensor at the cost of storing num_channels extra FP16 scales — negligible for most layers.

Model size and memory footprint

The arithmetic is unforgiving: INT8 halves weight memory versus FP16. A 7B parameter model occupies ~14 GB in FP16, ~7 GB in INT8, ~3.5 GB in INT4. KV cache, activation buffers, and framework overhead stay in FP16/BF16 during inference, so total VRAM savings are 30–40% for typical batch sizes, not 50%.

# Rough VRAM for 7B model, 4k context, batch=1
# FP16 weights: 14 GB + 1.2 GB KV + 0.5 GB activations ≈ 15.7 GB
# INT8 weights:  7 GB + 1.2 GB KV + 0.5 GB activations ≈ 8.7 GB

That 7 GB gap is the difference between fitting on a single RTX 3090/4090 (24 GB) versus needing model parallelism or CPU offload. For 70B models: FP16 ≈ 140 GB (8× A100 80 GB), INT8 ≈ 70 GB (4× A100 80 GB or 2× H100 80 GB with headroom).

Inference latency and throughput

INT8 wins on memory bandwidth — half the bytes to move from VRAM to compute units. On Hopper (H100) and Blackwell (B200), Tensor Cores natively accelerate INT8 GEMM with 2× throughput versus FP16. On Ampere (A100, RTX 30/40 series), INT8 tensor cores exist but software support in torch.compile, TensorRT-LLM, and vLLM is mature only for specific shapes.

Real-world throughput gains depend on bottlenecks:

Kernel launch overhead and quantization/dequantization (Q/DQ) nodes eat gains at small batch sizes. Fused Q/DQ kernels in TensorRT-LLM and vLLM’s int8 quantization mode mitigate this. Expect diminishing returns below batch=4 on GPU.

Quality degradation and calibration

Post-training quantization (PTQ) to INT8 typically costs 0.5–2% absolute accuracy on MMLU, GSM8K, and HumanEval for 7B–70B models. The damage concentrates in outlier channels — a few weights with magnitude 10–100× the mean. Two mitigations dominate:

SmoothQuant (Xiao et al., 2022) migrates outliers from weights to activations by scaling channels before quantization:

# SmoothQuant per-channel smoothing
alpha = 0.5  # typical
s = (act_max ** alpha) / (w_max ** (1 - alpha))
w_smooth = w * s
act_smooth = act / s

AWQ (Activation-aware Weight Quantization, Lin et al., 2023) protects salient channels using a small calibration set (128–512 sequences). No gradient updates, runs in minutes on a single GPU.

Quantization-aware training (QAT) recovers nearly all FP16 accuracy but requires full training pipeline access — rarely practical for proprietary models. For open weights, GPTQ (Frantar et al., 2022) and AWQ are the standard PTQ paths. Both need calibration data representative of your target domain; random WebText samples work for general chat, but code or multilingual workloads benefit from domain-matched calibration.

Deployment ergonomics

FP16 “just works” — load checkpoint, dispatch to device, generate. INT8 adds steps:

1. Calibration: Run 128–512 forward passes to collect activation statistics.
2. Quantize: Apply GPTQ/AWQ/SmoothQuant, produce INT8 checkpoint + scales.
3. Validate: Compare outputs on held-out prompts; check for NaNs or catastrophic failures.
4. Serve: Use a runtime that fuses Q/DQ (TensorRT-LLM, vLLM, llama.cpp).

llama.cpp handles this transparently via llama-quantize and runs INT8 on CPU/Metal/CUDA with zero external dependencies. vLLM’s quantization="int8" (via compressed-tensors format) and TensorRT-LLM’s quantize.py automate the pipeline for server deployments.

# llama.cpp quantization (CPU/Metal/CUDA)
llama-quantize model.f16.gguf model.q8_0.gguf q8_0

# vLLM INT8 serving
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization compressed-tensors \
  --dtype float16  # compute dtype; weights are INT8

Debugging INT8 failures is harder — silent accuracy drops, not crashes. Log per-layer quantization error (cosine similarity between FP16 and INT8 outputs) during validation.

Comparison table

Which to choose

Choose FP16 when:

• You have VRAM headroom (model fits with 20%+ margin).
• Quality is non-negotiable (medical, legal, code generation with strict evals).
• You lack calibration data representative of production traffic.
• You’re prototyping — skip quantization until latency/memory forces it.
• Running on older GPUs without INT8 tensor cores (pre-Ampere) where INT8 falls back to FP16 compute anyway.

Choose INT8 when:

• Model exceeds single-GPU VRAM at FP16 but fits at INT8 (7B on 16 GB, 70B on 80 GB×2).
• Serving high-throughput on H100/B200 where INT8 tensor cores double compute throughput.
• Deploying on CPU (llama.cpp, ONNX Runtime) or Apple Silicon (Core ML) — INT8 is 2–5× faster.
• Running at scale where 30% VRAM savings translates to fewer GPUs or larger batch sizes.
• You can allocate 30 minutes for calibration/quantization and have a validation set.

Default heuristic: Start FP16. Quantize to INT8 (AWQ or GPTQ) only when memory pressure or latency targets demand it. The calibration step is a one-time cost; the VRAM savings compound across every replica in production.