GPTQ vs AWQ: quantization method speed comparison

The decision between GPTQ and AWQ is usually framed around accuracy, but for serving systems the token throughput and memory footprint decide whether a model fits on a GPU. A grounded GPTQ vs AWQ speed comparison shows the runtime gap is smaller than community folklore suggests—both pack weights to 4 bits, and the real differentiator is kernel maturity on your target hardware. Below we break down the two methods across the dimensions that affect production inference.

How the two methods work

GPTQ

GPTQ approximates second-order information from a small calibration set to quantize each weight column with minimal reconstruction error. It produces a 4-bit weight matrix plus per-column scales and a small set of fp16 correction terms. The math is described in the GPTQ paper (Frantar et al., 2022) and implemented in AutoGPTQ and ExLlama. The quantizer walks columns left to right, applying an approximate inverse Hessian to compensate for rounding.

AWQ

AWQ (Activation-aware Weight Quantization) observes activation magnitudes to identify a small fraction of “salient” weights, keeping those at fp16 while quantizing the rest to 4-bit. It does not use reconstruction gradients; instead it applies a per-channel scaling before rounding. The result is a weight-only 4-bit tensor with mixed precision that maps cleanly to fused CUDA kernels. The calibration step only needs activation statistics, so it finishes in seconds rather than minutes.

Head-to-head dimensions

Capabilities

GPTQ supports 3, 4, and 8-bit variants and can quantize any linear layer in a transformer. AWQ targets 4-bit (and experimentally 3-bit) with mixed precision, and is primarily validated on decoder-only LLMs. For models under 13B, both recover >99% of fp16 perplexity on common benchmarks; AWQ often edges out GPTQ on instruction-tuned checkpoints because it protects activation-heavy channels. GPTQ’s act-order variant can recover more accuracy at the cost of a slightly more complex dequant path.

Price/cost model

Both methods cut VRAM by roughly 2.5–3x versus fp16, which is the dominant lever for cost. A 70B model that needs 140GB fp16 runs in 2x24GB or 1x48GB with 4-bit quant. The per-token cost scales with GPU-hours, and since both deliver similar decode speed, the economic difference is negligible. If you serve through a gateway such as n4n.ai, the quantization backend is hidden behind a model tag and you still get per-token metering regardless of whether the provider uses GPTQ or AWQ. The real savings come from avoiding the next GPU tier up.

Latency/throughput

This is the core of any GPTQ vs AWQ speed comparison. At 4-bit, weight-only quantization makes token generation memory-bandwidth bound. On A100/H100, both formats hit 70–90% of fp16 tokens/sec depending on batch size and kernel. The variance comes from the dequant path:

• GPTQ kernels (ExLlamaV2, AutoGPTQ cuda) fuse the inverse Hessian correction into the GEMM, giving strong single-stream latency.
• AWQ kernels (vLLM, TensorRT-LLM, llama.cpp) use a simpler scale+zero-point multiply, which is easier to optimize but may add a tiny prefill overhead.

In practice, for batch=1 the difference is often <5%; at large batches the gap narrows because compute is amortized. Neither method helps prefill compute-bound scenarios as much as fp8 would. Group size also matters: a 128-group quant adds more scale reads than 64-group, shaving a few percent off decode.

The theoretical speedup is bounded by memory bandwidth. With fp16 weights at 2 bytes/param and 4-bit at 0.5 bytes/param, you get a 4x reduction in weight fetch, but KV cache and attention stay fp16, so end-to-end gain is closer to 1.8–2.2x on decode-heavy workloads.

Ergonomics

GPTQ has the longest track record. A typical load pipeline:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-7b-GPTQ",
    device_map="auto",
    trust_remote_code=True,
)

Quantizing from scratch:

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
cfg = BaseQuantizeConfig(bits=4, group_size=128)
model = AutoGPTQForCausalLM.from_pretrained("meta-llama/Llama-2-7b", cfg)
model.quantize(calib_dataset)

AWQ requires a conversion step but ships ready-to-serve artifacts for vLLM:

from awq import AutoAWQForCausalLM
model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
model.quantize(calib_data, w_bit=4, q_group_size=128)

from vllm import LLM
llm = LLM(model="TheBloke/Llama-2-7b-AWQ", quantization="awq")

Both integrate with Hugging Face hubs. AWQ’s calibration is faster (no gradient approximation), but GPTQ’s tooling handles more architectures out of the box.

Ecosystem

GPTQ is native to ExLlama, text-generation-webui, and llama.cpp. AWQ is first-class in vLLM, TensorRT-LLM, and recent llama.cpp builds. If your stack is built on vLLM for continuous batching, AWQ is the path of least resistance. If you run a single-GPU ExLlama server, GPTQ will extract the last bit of latency. llama.cpp supports both via GGUF conversion, but the underlying mmq kernels treat them similarly.

Limits

GPTQ can degrade on tiny calibration sets (<128 samples) and requires the calibration loader to match the target sequence length. AWQ’s mixed precision means a naive PyTorch fallback (no custom kernel) runs slower than fp16 because of on-the-fly dequant. Both are post-training only; you cannot fine-tune in these formats without re-quantizing. Neither compresses the KV cache—that needs a separate method like quantized attention.

Comparison table

Calibration practicalities

For GPTQ, use at least 1,000 diverse sequences of the exact max context you will serve. A mismatch between calibration length and inference length hurts more than the bit width. AWQ needs far less data—a few hundred examples—because it only collects activation magnitudes. In both cases, avoid leaking test prompts into the calibration set; that artificially inflates accuracy.

Which to choose

Single-GPU local or hobby server

Use GPTQ with ExLlamaV2 if you want the fastest interactive chat on a 24GB card. The kernel is mature and the perplexity hit is invisible for 7–13B models. You can pull a pre-quantized GGUF or GPTQ repo and be serving in minutes.

Production serving with continuous batching

Pick AWQ on vLLM. The integration is maintained by the vLLM team, and the mixed-precision format avoids the calibration fragility that can surprise you at 3am when a new base model drops. The small prefill penalty disappears under batch load.

Accuracy-sensitive RAG or agents

Either works; run a quick eval on your validation set. If you see AWQ lag on a specific task, switch to GPTQ with a larger calibration corpus. The speed difference won’t break your SLA. Keep the same group size (128) for apples-to-apples comparison.

Multi-tenant gateway or API

Abstract the format away. Route by model id and let the platform handle fallback. In a setup like n4n.ai, the endpoint honors client routing directives and forwards provider cache-control hints, so your app sees one OpenAI-compatible interface whether the backing instance is GPTQ or AWQ. Automatic fallback covers the case where a specific quant variant is rate-limited or degraded.

The GPTQ vs AWQ speed comparison ultimately lands on “measure on your hardware.” Both are production-grade 4-bit methods; choose the one your serving kernel already optimizes for, and reserve the other as a drop-in fallback when a model variant isn’t published in your preferred format.