Measuring tokens per second quantized vs full precision on the same hardware is the fastest way to expose the real tradeoff in LLM serving. Quantization shrinks weight precision from FP16 to INT8 or INT4, trading some model fidelity for lower memory bandwidth and higher throughput. This article puts the two approaches side by side across the dimensions that matter when you ship.
What quantization actually changes
A transformer inference step is dominated by two costs: moving weights from HBM to compute and the matrix multiplies themselves. At FP16, a 70B model weighs ~140 GB, which forces tensor parallelism across multiple GPUs just to fit. Drop to INT4 and the same weights occupy ~35 GB, letting a single 80 GB A100 hold the model plus a large KV cache.
The arithmetic intensity shifts. Full-precision runs are often memory-bandwidth bound: you can’t feed the multipliers fast enough. Quantized runs reduce memory traffic but add dequantization overhead and may become compute bound at small batch sizes. The net effect on tokens per second quantized vs full precision is almost always positive for quantized, but the multiplier depends on batch size, context length, and whether the serving stack fuses dequant.
Head-to-head dimensions
Capabilities and output quality
Full-precision weights preserve the model’s trained distribution. On reasoning benchmarks (MMLU, GSM8K) and structured extraction, FP16/BF16 typically matches the reference numbers from the paper. INT8 is usually within a percent or two of those scores. INT4 (especially weight-only GPTQ/AWQ) can drop 2–5 points on hard reasoning and shows measurable degradation on code synthesis and long-context retrieval.
If your task is classification or draft generation, quantization is invisible. If you are running a multi-step agent that depends on exact arithmetic or nuanced instruction following, the gap is real.
Cost model
VRAM footprint drives cost. Fewer GPUs per model instance means more replicas per dollar and higher aggregate throughput. A quantized 70B can run on one node where full-precision needs four. That translates directly to per-token price drops of 2–4x in self-hosted clusters, assuming utilization is high.
The hidden cost is recomputation: if quantized output forces a human review or a second full-precision pass, you lose the savings. Meter your actual quality-adjusted cost, not just the raw GPU line item.
Latency and throughput
This is the core of tokens per second quantized vs full precision. At batch size 1, quantized models often deliver 1.3–2x the decode speed because weight fetch is halved or quartered. At large batch sizes serving many concurrent requests, the gap narrows because the system is saturating compute and KV cache bandwidth rather than weight loading.
Streaming time-to-first-token is usually better on quantized because model load and warmup are cheaper. Inter-token latency (the perceived “typing speed”) improves proportionally to the memory savings.
Ergonomics of serving
Full-precision checkpoints are the native format from model publishers. You load safetensors or sharded BF16 and go. Quantized formats require a conversion step: GGUF for llama.cpp, GPTQ/AWQ bins for vLLM, or FP8 kernels for TensorRT-LLM. That step is reproducible but adds a build artifact to your CI.
Hot-swapping between a full and quantized variant of the same base model is straightforward if your gateway keys on model name suffixes. Otherwise you maintain two deployment manifests.
Ecosystem and tooling
FP16 has universal support: PyTorch, vLLM, TGI, SGLang. Quantized paths are fragmented. llama.cpp dominates edge and CPU inference with GGUF. vLLM supports INT4/INT8 via Marlin kernels. TensorRT-LLM gives the best FP8 throughput but locks you to NVIDIA and a compile step.
Pick quantized only if your serving stack has first-class support for the specific scheme. A poorly tuned dequant kernel can erase the theoretical gain.
Hard limits
Quantization interacts badly with some architectures. Mixture-of-experts models (Mixtral, DeepSeek) have small active parameter counts, so weight memory is less of a bottleneck; quantization helps less. Very long context (128K+) inflates the KV cache, which stays in full precision in most stacks, capping the memory win. Some quantized formats drop rotary embedding scaling metadata, breaking long-context extrapolation.
Comparison table
| Dimension | Full-precision (FP16/BF16) | Quantized (INT8/INT4/FP8) |
|---|---|---|
| Capabilities | Matches reference accuracy; best on reasoning/code | INT8 near-parity; INT4 measurable drop on hard tasks |
| Cost model | High VRAM, more GPUs per model | 2–4x lower VRAM, fewer GPUs, cheaper per token |
| Throughput (tok/s) | Memory-bound; baseline | 1.3–2x at low batch; gap shrinks at high batch |
| Ergonomics | Native checkpoints, zero conversion | Requires GGUF/GPTQ/FP8 compile step |
| Ecosystem | Universal tooling | Fragmented; kernel-specific speedups |
| Limits | Context length only limit | MoE/long-context KV cache reduces benefit |
Requesting both from one client
The model name is the only switch most gateways expose. Using an OpenAI-compatible client:
from openai import OpenAI
client = OpenAI(base_url="https://your-gateway/v1", api_key="sk-...")
# full-precision request
fp = client.chat.completions.create(
model="meta-llama/llama-3-70b-instruct",
messages=[{"role": "user", "content": "Summarize the S3 billing doc."}],
max_tokens=256,
)
# quantized variant (naming varies by provider)
q = client.chat.completions.create(
model="meta-llama/llama-3-70b-instruct-q4",
messages=[{"role": "user", "content": "Summarize the S3 billing doc."}],
max_tokens=256,
)If you front your stack with a router that honors client routing directives, you can pin the quantized ID for bulk jobs and fall back to the full-precision ID on validation failure without rewriting the call site.
Throughput measurement without fake numbers
You can measure tokens per second quantized vs full precision yourself with a tiny loop:
curl -s https://your-gateway/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-d '{"model":"meta-llama/llama-3-8b-instruct-q8","messages":[{"role":"user","content":"Write 500 words on TCP."}],"max_tokens":800}' \
-o /dev/null -w "%{time_total}\n"Divide generated tokens by time_total minus time-to-first-token to get decode tok/s. Run at batch 1 and at concurrent 32 to see the curve.
Which to choose
High-volume, low-stakes generation (summarization, draft replies, log classification): use INT4/INT8. The tokens per second quantized vs full precision win pays for the slight quality loss, and the VRAM savings let you pack more replicas.
Agentic workflows, code gen, financial/legal extraction: stay on FP16/BF16. The cost of a wrong tool call or missed clause exceeds the GPU savings. If cost is still prohibitive, try INT8 first—it is usually safe.
Mixed traffic with strict SLOs: deploy both and route by task. A gateway that honors client routing directives—such as n4n.ai—lets you send "route": {"prefer": "quantized"} for bulk jobs and strip the hint for premium requests, while per-token metering keeps the finance team happy.
Edge or single-GPU deployments: quantized is the only option that fits. GGUF on llama.cpp turns a 70B into a laptop workload at the cost of slower decode than a datacenter GPU.
Quantization is not a free lunch, but for throughput-bound serving it is the highest-leverage knob you have. Measure on your own prompts, at your own batch size, before committing.