Topic
Quantization Formats: GGUF, GPTQ, AWQ & INT4/INT8
8 posts on quantization formats: gguf, gptq, awq & int4/int8 — part of glossary on the n4n AI blog.
What is GPTQ quantization? How it works
GPTQ quantization explained: how the layer-wise Hessian-based method compresses LLMs to 4-bit with minimal accuracy loss, plus practical trade-offs and misconceptions.
What is GGUF? The format behind llama.cpp models
GGUF explained — the single-file model format powering llama.cpp, with quantization details, metadata structure, and practical usage patterns for local inference.
What is AWQ? Activation-aware weight quantization
AWQ quantizes LLM weights to INT4 by protecting salient channels identified through activation statistics, preserving accuracy better than GPTQ or naive PTQ.
How to choose a quantization format for local inference
A practical decision framework for picking GGUF, GPTQ, AWQ, or INT4/INT8 quantization for local LLM inference, with verification steps and hardware-specific guidance.
GGUF vs GPTQ vs AWQ: which quantization format to use
Compare GGUF, GPTQ, and AWQ quantization formats across hardware support, inference speed, model compatibility, and deployment ergonomics — with a clear verdict for each use case.
GGUF vs GGML: what changed and why
Technical comparison of GGUF vs GGML quantization formats — what changed, why it matters for llama.cpp inference, and which to use for local LLM deployment.
GGUF quantization levels: Q4_K_M vs Q5_K_M vs Q8_0
Compare Q4_K_M, Q5_K_M, and Q8_0 GGUF quantization levels across memory, speed, quality, and hardware fit — with a clear verdict for each use case.
AWQ vs GPTQ: which is faster for serving LLMs?
AWQ vs GPTQ compared on quantization quality, GPU kernel speed, model availability, and serving stack support — with a verdict for each deployment scenario.
More topics in glossary
- Structured Outputs & JSON Mode19
- AI Agents Fundamentals12
- Hallucination in LLMs11
- Sampling Parameters: Top-p, Top-k & Penalties11
- Context Window & Context Length10
- Fine-Tuning Fundamentals9
- Foundation Models: Base vs Instruct vs Chat9
- Model Families & Naming Conventions: GPT-5, Claude, Gemini 3, Llama 4, Mistral, DeepSeek, Qwen, Grok9
- Grounding & Fact-Checking in AI8
- LLM Benchmarks: MMLU, HumanEval, SWE-bench & GPQA8
- Max Tokens, Stop Sequences & Output Truncation8
- Chain-of-Thought Prompting7