INT4 quantization speed gains on consumer GPUs

INT4 quantization consumer GPU speed gains are not a free lunch, but on bandwidth-bound inference they are the single highest-leverage change you can make to a consumer box. Cutting weights from FP16 to 4-bit integers shrinks the model footprint by ~4x and, because most decoder-only transformers spend their time shuffling weights from VRAM rather than doing dense math, that reduction directly converts to higher token throughput on hardware like an RTX 4090 or 3090. The caveat is that integer quantization introduces accuracy loss and requires mature kernels; you ship it only after measuring both axes.

The bandwidth wall on consumer silicon

Consumer GPUs ship with massive compute throughput but comparatively thin memory buses. A RTX 4090 delivers ~1 TB/s of GDDR6X bandwidth; a datacenter A100 offers 2 TB/s with HBM2e. Regardless, for autoregressive generation the dominant cost is weight fetch, not MACs. Each forward pass reads the entire parameter set per token. At FP16, a 13B model weighs ~26 GB, exceeding the 24 GB framebuffer of most consumer cards. At INT4, it drops to ~6.5 GB, fitting comfortably with KV cache headroom.

This is why INT4 quantization consumer GPU speed improvements scale with model size relative to VRAM. If the model already fits in FP16, the speedup is smaller—you still save bandwidth, but kernel launch overhead and dequantization compute eat part of the win.

What INT4 actually changes in the pipeline

INT4 replaces each 16-bit float weight with a 4-bit integer plus a per-block scale factor. The GPU loads packed 4-bit values, unpacks them in registers, and executes INT4 tensor core matmuls or converts to FP16 for compute. NVIDIA tensor cores have supported INT4 dot products since Turing; Ada Lovelace increased that throughput further. AMD RDNA3 also exposes INT4 paths via ROCm.

The theoretical memory reduction is 4x. Real token-rate gains land between 1.5x and 3x on consumer parts for models that were previously VRAM-constrained or bandwidth-saturated. You do not get 4x because:

• Dequantization adds ALU work.
• KV cache stays in higher precision.
• Small batch sizes underutilize tensor cores.

KV cache and mixed precision

Only weights are typically quantized. The KV cache remains FP16 or FP8 because its size scales with sequence length and batch, not parameter count. On long-context workloads the cache can become the bandwidth bottleneck, blunting INT4 weight savings. If you serve 32k contexts on a consumer GPU, measure carefully—INT4 helps less than you expect.

Implementation paths that actually work

Hugging Face + bitsandbytes

For PyTorch workflows, bitsandbytes provides in-place 4-bit loading. The default is NF4 (a normalized float), but the config below forces a pure integer path where supported.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="int4",  # NF4 is default; int4 available in recent versions
    bnb_4bit_compute_dtype="fp16",
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-13b-chat-hf",
    quantization_config=bnb_config,
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-chat-hf")

This gets you a model that runs on a single 24 GB card. Expect ~30-40% of the FP16 token rate if the FP16 model could be sharded across two GPUs, but ~2x the rate of a CPU-offload FP16 baseline on the same card.

llama.cpp / GGML

For maximum speed on consumer GPUs, the GGML backend with INT4 weights is hard to beat. The quantization step is explicit:

# Convert and quantize to q4_0 (INT4 symmetric)
./llama-quantize ./models/llama-13b-f16.gguf ./models/llama-13b-q4_0.gguf q4_0
# Run inference on GPU layer 0..all
./llama-cli -m ./models/llama-13b-q4_0.gguf -p "Explain INT4 quantization" -n 256 -ngl 99

The -ngl 99 flag offloads all layers to the GPU. On a 4090, a 13B q4_0 model generates north of 60 tokens/sec, versus ~25 tokens/sec for the same model in FP16 partially offloaded.

GPTQ and AWQ

Training-free post-training quantization libraries like GPTQ and AWQ produce INT4 checkpoints with minimal perplexity drift. They use calibration sets and mixed precision for sensitive layers. If you need reproducible INT4 weights across serving stacks, these formats are more portable than bitsandbytes’ runtime quant.

Benchmarking without fooling yourself

Measure token throughput at the batch size and context length you will actually serve. A micro-benchmark with a 1-token prompt and max batch 1 hides the KV cache bandwidth tax. Use a realistic prompt and sequence:

import time
from transformers import pipeline

# Assume model loaded in 4bit as above
pipe = pipeline("text-generation", model=model, tokenizer=tok, device=0)
prompt = "Summarize the tradeoffs of 4-bit quantization." * 20  # ~120 tokens
start = time.time()
out = pipe(prompt, max_new_tokens=200, do_sample=False)
elapsed = time.time() - start
tokens_generated = 200  # approximate
print(f"{(tokens_generated)/elapsed:.1f} tok/s")

Run the same script against an FP16 reference on identical hardware. If the FP16 model doesn’t fit, compare against a CPU-offload baseline or a smaller model—but label it honestly. INT4 quantization consumer GPU speed claims must come from your own workload, not vendor charts.

Accuracy: where INT4 hurts

INT4 is not lossless. Perplexity on WikiText typically rises by 0.1–0.5 for 13B+ models, and more for 7B classes. Sensitive tasks—tool calling, numeric reasoning, code synthesis—degrade earlier than open-domain chat. Calibration matters: a poorly calibrated INT4 7B can lose several points on MMLU-style evals, while a well-tuned AWQ INT4 70B stays within 1% of FP16 on many reported benchmarks.

Rule of thumb: the larger the model, the safer INT4. A 70B INT4 on two consumer GPUs beats a 13B FP16 on one for both speed and quality in many cases.

Deployment topology

If you serve multiple quant levels, route by workload. A gateway like n4n.ai that honors client routing directives can pin INT4-quantized variants to consumer GPU pools while reserving full-precision endpoints for tasks needing higher fidelity. This keeps cost down without forcing every request through the accuracy penalty.

When you self-host, consider that INT4’s smaller footprint lets you colocate multiple models on one card. A 24 GB GPU can hold a 13B INT4 plus a 7B INT4 with room for KV cache, enabling parallel serving that would be impossible in FP16.

Tradeoffs summary

Pros:

• Fits larger models in consumer VRAM.
• 1.5–3x token throughput on bandwidth-bound loads.
• Lower power per token.

Cons:

• Accuracy loss, worst on small models.
• Kernel support fragmented (CUDA vs ROCm vs CPU).
• Dequant overhead reduces gains at large batch.

Takeaway

Ship INT4 quantization on consumer GPUs when your model is VRAM-constrained or you have measured a bandwidth bottleneck at production batch sizes, and only after validating perplexity and task accuracy on a real calibration set. For 13B+ parameters on a single 24 GB card, it is the default choice. For 7B or smaller models where FP16 already fits, stay in FP16 unless you need the extra throughput for high concurrency. INT4 quantization consumer GPU speed wins are real, but they belong to engineers who benchmark before believing them.