What is AWQ? Activation-aware weight quantization

Activation-aware weight quantization (AWQ) is a post-training quantization method that compresses LLM weights to 4-bit integers while preserving model quality by protecting the most important weight channels. Unlike GPTQ or naive round-to-nearest approaches, AWQ identifies salient channels using activation statistics from a small calibration set, then applies per-channel scaling that keeps quantization error low where it matters most. The result is INT4 models that run faster on consumer GPUs with minimal accuracy degradation.

How AWQ works

The core insight behind AWQ is that not all weight channels contribute equally to model output. A small fraction of channels — typically 1% — carry disproportionate activation magnitude. Quantizing these aggressively destroys model quality. AWQ protects them through a three-stage process.

Stage 1: Activation statistics collection

AWQ runs a small calibration dataset (128–512 sequences) through the FP16 model and records per-channel activation maxima. For each linear layer with weight matrix $W \in \mathbb{R}^{d_{out} \times d_{in}}$, it computes:

# Pseudocode for AWQ calibration
def collect_activation_stats(model, calib_data, n_samples=128):
    stats = {}
    hooks = []
    
    def hook_fn(module, input, output, name):
        # input[0] shape: [batch, seq, d_in]
        # We care about per-input-channel max magnitude
        x = input[0].abs().amax(dim=(0, 1))  # [d_in]
        stats[name] = x
    
    for name, module in model.named_modules():
        if isinstance(module, nn.Linear):
            hooks.append(module.register_forward_hook(
                lambda m, i, o, n=name: hook_fn(m, i, o, n)
            ))
    
    with torch.no_grad():
        for batch in calib_data[:n_samples]:
            model(batch)
    
    for h in hooks:
        h.remove()
    return stats

This yields a salience metric per input channel: channels with consistently large activations are “salient.”

Stage 2: Per-channel scaling

For each output channel $j$, AWQ computes a scaling factor $s_j$ that minimizes quantization error on salient input channels. The optimization solves:

$$s_j^* = \arg\min_s | W_j - \text{quantize}(W_j \cdot s) / s |_2^2 \cdot \text{mask}_j$$

where $\text{mask}_j$ weights input channels by their activation magnitude. In practice, AWQ uses a grid search over scaling factors (typically 0.5–2.0) and picks the one that minimizes weighted MSE on the salient subset.

def awq_scale_search(weight, act_scales, n_grid=20):
    """
    weight: [d_out, d_in]
    act_scales: [d_in] — per-input-channel activation max
    Returns: best_scale [d_out]
    """
    # Identify salient channels (top 1% by activation magnitude)
    k = max(1, act_scales.numel() // 100)
    salient_idx = act_scales.topk(k).indices
    salient_mask = torch.zeros_like(act_scales, dtype=torch.bool)
    salient_mask[salient_idx] = True
    
    best_scales = torch.ones(weight.shape[0], device=weight.device)
    best_errors = torch.full((weight.shape[0],), float('inf'))
    
    for scale in torch.linspace(0.5, 2.0, n_grid):
        # Quantize with this scale
        w_scaled = weight * scale
        w_q = torch.round(w_scaled.clamp(-8, 7))  # INT4 symmetric
        w_deq = w_q / scale
        
        # Weighted MSE on salient channels only
        error = ((weight - w_deq) ** 2 * salient_mask.float()).sum(dim=1)
        
        update = error < best_errors
        best_errors[update] = error[update]
        best_scales[update] = scale
    
    return best_scales

Stage 3: Quantization and packing

With optimal scales in hand, AWQ quantizes each output channel independently:

def awq_quantize(weight, scales):
    """
    weight: [d_out, d_in] fp16
    scales: [d_out] fp32
    Returns: int4 packed weights, scales for dequant
    """
    d_out, d_in = weight.shape
    w_scaled = weight * scales.unsqueeze(1)
    w_q = torch.round(w_scaled.clamp(-8, 7)).to(torch.int8)  # INT4 range
    
    # Pack two int4 values per int8 (4-bit packing)
    w_packed = (w_q[:, ::2] & 0xF) | ((w_q[:, 1::2] & 0xF) << 4)
    
    return w_packed.to(torch.uint8), scales

The packed INT4 weights and per-output-channel scales are all that’s needed for inference. Dequantization at runtime is a single multiply per output channel.

Why AWQ matters for inference

AWQ addresses the central tension in LLM deployment: model size versus quality. FP16 weights at 7B parameters consume ~14 GB VRAM. INT4 brings that to ~3.5 GB, enabling 7B models on 8 GB consumer cards and 70B models on 24 GB multi-GPU setups.

The quality preservation is measurable. On LLaMA-2-7B, AWQ INT4 typically loses 0.5–1.5% absolute accuracy on MMLU versus FP16, compared to 3–5% for naive RTN (round-to-nearest) and 1–2% for GPTQ. The gap widens at smaller model sizes where each parameter carries more signal.

Latency improvements are equally significant. INT4 GEMM kernels on modern GPUs (Ampere, Hopper, Ada) achieve 2–4× throughput versus FP16 for memory-bound workloads. On an RTX 4090, a 7B AWQ model can sustain 80–120 tokens/second at batch size 1, versus 30–50 for FP16.

Concrete example: Quantizing LLaMA-3-8B

Here’s a complete workflow using the reference awq library:

# Install
pip install autoawq

# Quantize
python -m awq.entry --model_path meta-llama/Meta-Llama-3-8B \
    --quant_path ./Llama-3-8B-AWQ \
    --w_bit 4 --q_group_size 128 \
    --calib_data wikitext2 --n_calib 128 \
    --max_seq_len 2048

Key parameters:

• w_bit 4: Target INT4 quantization
• q_group_size 128: Group size for per-group scaling (128 is standard; 64 improves quality at slight size cost)
• calib_data wikitext2: Calibration dataset — domain-matched data helps
• n_calib 128: Number of calibration sequences — 128 is sufficient; more yields diminishing returns

The output directory contains:

Llama-3-8B-AWQ/
├── config.json          # Model config with quantization metadata
├── model.safetensors    # Packed INT4 weights + scales
└── tokenizer.*          # Unchanged tokenizer files

Loading for inference with autoawq or llama.cpp (via GGUF conversion):

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model = AutoAWQForCausalLM.from_quantized("./Llama-3-8B-AWQ", fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained("./Llama-3-8B-AWQ")

# Generate
tokens = tokenizer("The future of AI is", return_tensors="pt").input_ids.cuda()
output = model.generate(tokens, max_new_tokens=64)
print(tokenizer.decode(output[0]))

With fuse_layers=True, the library fuses LayerNorm + QKV projections and applies AWQ’s pre-computed scales, eliminating runtime dequantization overhead for those ops.

AWQ versus GPTQ versus GGUF

AWQ wins on pure GPU throughput and quality at INT4. GPTQ is comparable but slower to quantize (Hessian computation). GGUF dominates CPU and Apple Silicon inference but lags on discrete GPU throughput due to less optimized INT4 kernels.

For cloud GPU deployments where you control the hardware, AWQ or GPTQ are the right choices. For edge, mobile, or heterogeneous environments, GGUF’s portability wins.

Common misconceptions

“AWQ requires no calibration data”

False. AWQ needs a representative calibration set to compute activation statistics. Random noise produces poor scales. The calibration data should match your target domain — code for coding models, chat logs for chat models, etc. 128 sequences of 2048 tokens is the practical minimum; 512 is safer.

“AWQ is just per-channel scaling”

Per-channel scaling exists in many quantizers. AWQ’s distinction is the activation-aware weighting of the scaling objective. Standard per-channel scaling minimizes uniform MSE. AWQ minimizes MSE weighted by activation magnitude, which preserves the channels that actually drive model behavior.

“Group size doesn’t matter much”

Group size (typically 64 or 128) controls how many input channels share a scale. Smaller groups = more scales = better quality = larger model. At INT4, group size 64 adds ~0.5% model size but can recover 0.5–1% accuracy on difficult tasks. For 7B+ models, 128 is a good default. For 3B and below, use 64.

“AWQ works equally well for all layers”

Attention projections (q_proj, k_proj, v_proj, o_proj) and MLP gates (gate_proj, up_proj, down_proj) have different sensitivity profiles. AWQ applies the same algorithm to all, but the effective protection varies. Down-projection and o_proj tend to be most sensitive; gate_proj often tolerates aggressive quantization. Some practitioners selectively skip quantization on the most sensitive layers (e.g., keep down_proj at FP16) for a 10–15% size increase but near-FP16 quality.

“You can quantize any model with AWQ and expect good results”

Models trained with quantization-aware training (QAT) or with smooth quantization (SmoothQuant) quantize better. Standard pretrained models vary: LLaMA-family quantizes well; some MoE models show larger degradation due to expert specialization. Always evaluate on your downstream tasks.

Practical deployment notes

Kernel requirements: AWQ INT4 needs custom kernels for the packed GEMM. autoawq ships Triton kernels for Ampere+. For Hopper (H100), use TensorRT-LLM’s AWQ path which leverages native INT4 tensor cores. On AMD, vLLM with rocm backend supports AWQ via hipBLASLt.

KV cache: AWQ quantizes weights only. KV cache remains FP16 or BF16 by default. For further memory savings, combine AWQ with KV cache quantization (FP8 or INT8) — supported in vLLM 0.4+ and TensorRT-LLM.

Merging LoRA: If you have LoRA adapters, merge them before AWQ quantization:

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B", torch_dtype=torch.float16)
merged = PeftModel.from_pretrained(base, "./my-lora").merge_and_unload()
merged.save_pretrained("./merged-fp16")
# Then run AWQ on ./merged-fp16

Quantizing a base model then applying LoRA at inference adds dequantization overhead and typically degrades quality.

Evaluation: Don’t trust perplexity alone. Run your actual eval harness — MMLU, GSM8K, HumanEval, or your domain benchmarks. AWQ can preserve perplexity while degrading reasoning or code generation. A 0.1 perplexity delta can mask a 5% pass@1 drop on HumanEval.

When to choose AWQ

Choose AWQ when:

• Deploying on NVIDIA GPUs (Ampere, Ada, Hopper)
• Targeting INT4 for maximum throughput/VRAM savings
• You have a calibration dataset matching your domain
• Quality at INT4 is critical (beats GPTQ marginally, beats RTN significantly)

Choose GPTQ when:

• You already have a GPTQ pipeline and the quality delta doesn’t justify migration
• Quantizing on CPU-only infrastructure (GPTQ’s Hessian computation is CPU-friendly)

Choose GGUF when:

• Targeting CPU, Apple Silicon, or mixed hardware
• You need a single artifact that runs everywhere
• Model size is small enough that CPU inference is viable

AWQ has become the de facto standard for high-throughput GPU inference at INT4. The activation-aware scaling is a simple idea with outsized impact — it turns what would be a 3–5% quality hit into a 0.5–1.5% hit, which is often the difference between “usable” and “broken” for production workloads.