AWQ vs GPTQ: which is faster for serving LLMs?

When you’re choosing between AWQ vs GPTQ for production LLM serving, the decision rarely comes down to theoretical quality metrics. It comes down to which format your inference engine actually accelerates, whether you can find pre-quantized weights for your model, and how much VRAM you save per request. Both methods compress weights to INT4 (or INT3/INT8) without retraining, but they optimize for different things and the ecosystem has settled around AWQ for high-throughput GPU serving.

What each format actually does

GPTQ (Gradient-based Post-Training Quantization) applies layer-wise quantization using second-order information from the Hessian matrix. It processes one layer at a time, solving a weighted least-squares problem to minimize the reconstruction error of the layer’s output given quantized weights. The original paper uses a greedy coordinate descent with Cholesky decomposition of the Hessian. In practice, AutoGPTQ implements this with grouped quantization — typically 128 weights share a scale and zero-point — which keeps the metadata overhead low.

AWQ (Activation-aware Weight Quantization) takes a different tack: it identifies “salient” weight channels by observing activation magnitudes on a small calibration set, then protects those channels during quantization. The insight is that not all weights contribute equally to output quality; channels that consistently see large activations matter more. AWQ applies per-channel scaling (not grouped) and uses a learned per-channel scaling factor that gets folded into the next layer’s weights, avoiding runtime dequantization overhead.

Both produce INT4 weights with FP16 activations at inference time. The difference is in how they pick which weights get higher precision and how the quantization parameters are structured.

Quantization approach and quality

At equivalent bit-widths, AWQ typically preserves perplexity better than GPTQ, especially below 4-bit. The activation-aware scaling means the quantization error distributes more evenly across the network. GPTQ’s Hessian-based approach is theoretically sound but sensitive to calibration data quality and the damping factor used during Cholesky decomposition.

# AutoGPTQ quantization example
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False,  # act-order: reorder weights by activation magnitude
)

model = AutoGPTQForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantize_config=quantize_config,
)
model.quantize(calibration_dataset)
model.save_quantized("./llama-2-7b-gptq-4bit")

# AutoAWQ quantization example
from awq import AutoAWQForCausalLM

quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",  # or "GEMV" for batch=1
}

model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model.quantize(tokenizer, quant_config=quant_config, calib_data=calibration_dataset)
model.save_quantized("./llama-2-7b-awq-4bit")

Notice the version parameter in AWQ: GEMM kernels assume batch > 1 and use matrix-matrix multiplication; GEMV kernels optimize for batch=1 (matrix-vector). This distinction matters for serving. GPTQ doesn’t have this split — its kernels handle both, but the desc_act (activation-order) flag changes weight layout and affects kernel compatibility.

Serving performance: latency and throughput

This is where AWQ pulls ahead for most GPU deployments. The per-channel scaling in AWQ maps cleanly to the w4a16 GEMM kernels in vLLM, TensorRT-LLM, and MLC-LLM. These kernels fuse dequantization with the matrix multiply, keeping weights in INT4 in shared memory and only expanding to FP16 during the multiply-accumulate. The result: AWQ models often run 1.3–1.5× faster than equivalent GPTQ models at the same batch size on H100/A100.

GPTQ’s grouped quantization (128 weights per scale) requires a different kernel pattern. The scales must be broadcast across the group during dequantization, which adds instructions and register pressure. vLLM supports GPTQ via the Marlin kernel, but Marlin is optimized for specific tile sizes and doesn’t always saturate the tensor cores as cleanly as the AWQ kernels do.

# vLLM serving both formats
vllm serve TheBloke/Llama-2-7B-AWQ --quantization awq --max-model-len 4096
vllm serve TheBloke/Llama-2-7B-GPTQ --quantization gptq --max-model-len 4096

TensorRT-LLM has first-class AWQ support with its weight_only_quant plugin. GPTQ support exists but often requires converting to AWQ format first or using the older quantize.py workflow. MLC-LLM and llama.cpp both support both formats, but their AWQ paths are more heavily optimized for mobile and embedded GPUs.

For batch=1 latency (GEMV regime), the gap narrows. AWQ’s GEMV kernels and GPTQ’s ExLlamaV2 kernels both achieve near-memory-bandwidth-bound performance. If your workload is strictly single-user, interactive chat, either format works. If you’re batching requests — which you should be for any production API — AWQ’s GEMM kernels win.

Hardware support and kernel availability

The kernel story is the practical differentiator. If you’re on NVIDIA GPUs with vLLM or TensorRT-LLM, AWQ is the path of least resistance. If you’re on AMD or Intel, both are roughly equivalent — check the specific engine’s release notes for your hardware generation. For CPU-only or Apple Silicon, llama.cpp treats both formats identically; the quantization method matters less than the quantization library’s calibration quality.

Model availability and ecosystem

Hugging Face hosts roughly 3× as many AWQ models as GPTQ models at this point. The Bloke’s repository (now maintained by the community) standardized on AWQ. Major model releases — Llama-3, Qwen2, Mistral-Nemo, Nemotron — typically ship AWQ quantizations within days. GPTQ quantizations still appear but often lag by weeks.

AutoAWQ and AutoGPTQ are both actively maintained. AutoAWQ added support for Llama-3’s grouped-query attention and Qwen2’s tied embeddings faster. AutoGPTQ added desc_act=True (activation-ordered weights) which improves quality but breaks compatibility with Marlin kernels unless you also use --act-order at serve time — a footgun that’s bitten several teams.

// AWQ config saved with model (config.json)
{
  "quantization_config": {
    "quant_method": "awq",
    "bits": 4,
    "group_size": 128,
    "zero_point": true,
    "version": "GEMM",
    "modules_to_not_convert": ["lm_head"]
  }
}

// GPTQ config saved with model (quantize_config.json)
{
  "bits": 4,
  "group_size": 128,
  "damp_percent": 0.01,
  "desc_act": false,
  "static_groups": false,
  "sym": true,
  "true_sequential": true,
  "model_name_or_path": "meta-llama/Llama-2-7b-hf"
}

The modules_to_not_convert field in AWQ is practical: keeping lm_head in FP16 costs ~0.5% VRAM and often recovers the last 0.1–0.2 perplexity points. GPTQ configs don’t standardize this; you have to handle it in the quantization script.

Operational ergonomics

Quantizing a model yourself? AWQ is faster to quantize. A 7B model takes ~15 minutes on an A100 with AutoAWQ; GPTQ takes ~45 minutes because of the Hessian computation. The calibration dataset matters for both — 128 sequences of 2048 tokens is the standard — but AWQ is less sensitive to calibration distribution shift.

Serving both formats in the same stack is straightforward with vLLM:

from vllm import LLM, SamplingParams

# AWQ model
llm_awq = LLM(
    model="TheBloke/Llama-2-7B-AWQ",
    quantization="awq",
    max_model_len=4096,
    gpu_memory_utilization=0.9,
)

# GPTQ model
llm_gptq = LLM(
    model="TheBloke/Llama-2-7B-GPTQ",
    quantization="gptq",
    max_model_len=4096,
    gpu_memory_utilization=0.9,
)

One operational gotcha: GPTQ models quantized with desc_act=True (activation ordering) require the serving engine to know the activation order. vLLM’s Marlin kernel handles this automatically if the model config includes quantization_config.desc_act: true. But if you quantized with AutoGPTQ v0.5+ and serve with an older vLLM, you’ll get silent quality degradation. AWQ doesn’t have this version skew problem — the version field (GEMM/GEMV) is explicit and backward compatible.

Comparison table

Which to choose

Choose AWQ if:

• You’re serving on NVIDIA GPUs with vLLM or TensorRT-LLM and care about throughput. The kernel advantage is real and compounds at scale.
• You want the widest selection of pre-quantized models on Hugging Face. Someone has already quantized your model correctly.
• You quantize models yourself and value iteration speed. 15 minutes vs 45 minutes per experiment adds up.
• You’re building a new deployment today. The ecosystem momentum is behind AWQ.

Choose GPTQ if:

• You’re on AMD GPUs with vLLM and the Marlin kernel is your best option (check current ROCm release notes — this changes quarterly).
• You’re strictly batch=1 on consumer GPUs and ExLlamaV2’s GEMV kernels outperform AWQ’s GEMV for your specific model size.
• You have an existing GPTQ model pipeline that works and the migration cost exceeds the throughput gain.
• You need INT3 or INT2 quantization; AutoGPTQ supports lower bit-widths more maturely than AutoAWQ.

Choose neither (use GGUF/llama.cpp) if:

• You’re deploying on CPU, Apple Silicon, or edge devices without CUDA/ROCm.
• You need a single file that runs everywhere without a serving engine.
• Your batch size is 1 and latency matters more than throughput.

The industry has converged on AWQ for high-throughput GPU serving. The kernels are faster, the models are easier to find, and the quantization loop is tighter. GPTQ remains a solid fallback for specific hardware/engine combinations, but it’s no longer the default choice for new deployments.