Knowledge distillation vs quantization: the difference

If you’re compressing LLMs for production, you’ll eventually hit the knowledge distillation vs quantization decision. Both shrink models. Both cut latency. But they operate at different layers of the stack, impose different constraints, and fail in different ways. This post breaks down the trade-offs so you can pick the right tool — or combine them — without learning the hard way.

What each technique actually does

Quantization reduces the numerical precision of model weights and activations. You take a model trained in FP16 or BF16 and map its parameters to INT8, INT4, or even lower bit-widths. The architecture stays identical; only the representation changes. Post-training quantization (PTQ) does this without gradient updates. Quantization-aware training (QAT) fine-tunes with simulated low-precision arithmetic to recover accuracy.

Knowledge distillation trains a smaller student model to mimic a larger teacher. The student learns from the teacher’s logits (soft labels), hidden states, or attention distributions — not just the ground-truth tokens. The student architecture can differ entirely: fewer layers, smaller hidden dimension, different attention pattern. You’re not compressing the same model; you’re building a new one that approximates the teacher’s behavior.

# Quantization: same architecture, lower precision
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B", torch_dtype=torch.float16)
# PTQ example with bitsandbytes
model_4bit = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8B",
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
)

# Distillation: different architecture, trained from teacher logits
# Student config: 4 layers instead of 32, 2048 hidden instead of 4096
student_config = LlamaConfig(
    num_hidden_layers=4,
    hidden_size=2048,
    intermediate_size=5504,
    num_attention_heads=16,
)
student = LlamaForCausalLM(student_config)

# Training loop minimizes KL divergence between teacher and student logits
def distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):
    soft_loss = F.kl_div(
        F.log_softmax(student_logits / temperature, dim=-1),
        F.softmax(teacher_logits / temperature, dim=-1),
        reduction="batchmean",
    ) * (temperature ** 2)
    hard_loss = F.cross_entropy(student_logits.view(-1, vocab_size), labels.view(-1))
    return alpha * soft_loss + (1 - alpha) * hard_loss

Capabilities and quality retention

Quantization preserves the original model’s capabilities — reasoning, coding, multilingual support — up to a point. INT8 is nearly lossless for most tasks. INT4 degrades noticeably on complex reasoning and long-context tasks. Below INT4, you hit a quality cliff unless you use advanced techniques like GPTQ, AWQ, or QAT with careful calibration data.

Distillation can exceed the teacher on specific distributions if the student overfits to the teacher’s outputs on your target domain. But it typically loses broad capabilities: the student won’t know facts the teacher knew unless those facts appear in the distillation data. It also inherits the teacher’s biases and hallucinations, sometimes amplified.

Latency and throughput

Quantization wins on raw inference speed per parameter. INT4 kernels on modern GPUs (Hopper, Blackwell) and Apple Silicon achieve 2-4x throughput over FP16 for memory-bound workloads. The model stays the same size in layers, so you still pay the full depth penalty: 32-layer models have 32 sequential attention blocks regardless of precision.

Distillation reduces depth and width. A 4-layer student from a 32-layer teacher cuts sequential operations by 8x. That translates to lower latency (time to first token) and higher throughput at small batch sizes where depth dominates. But the student’s smaller hidden size reduces arithmetic intensity, so you lose some hardware utilization at large batch sizes.

# Rough latency comparison on H100, batch=1, 2k context
# Llama-3-8B FP16:     ~45 ms/token
# Llama-3-8B INT4:     ~18 ms/token  (2.5x speedup)
# Distilled 1.3B student: ~8 ms/token  (5.6x speedup)
# Distilled 1.3B INT4:  ~4 ms/token  (11x speedup)

The combination — distill then quantize — often dominates either alone. A 1.3B student at INT4 fits in 1 GB VRAM and runs faster than any 8B quantized model.

Cost model: compute, engineering, and operational

Quantization is cheap to apply. PTQ takes minutes on a single GPU. QAT costs a fine-tuning run (hours to days). No data curation beyond a calibration set. Engineering effort: integrate a quantization library (bitsandbytes, GPTQ, AWQ, or TensorRT-LLM), handle calibration, validate quality. Operational cost: you serve the same model artifact, just with a different loader.

Distillation is expensive. You need:

• Teacher inference compute to generate soft labels (or logits) over your training corpus
• A full pre-training or continued pre-training run for the student
• Curriculum design: what data, what loss weights, what temperature schedule
• Evaluation infrastructure to catch capability collapse

Engineering effort is higher: custom training loops, distributed training for the student, logit storage/streaming pipelines. Operational cost: you now maintain a separate model lineage. When the teacher updates, you must re-distill.

# Rough cost comparison for an 8B -> 1B compression
# Quantization (PTQ + validation):
#   GPU-hours: ~2 (calibration + eval)
#   Engineer-weeks: 1-2
#   Recurring: near zero

# Distillation:
#   GPU-hours: ~5,000-20,000 (student training)
#   Teacher inference: ~1,000 GPU-hours (logit generation)
#   Engineer-weeks: 6-12
#   Recurring: re-distill on teacher updates

Ergonomics and ecosystem maturity

Quantization tooling is mature and standardized. bitsandbytes integrates with Hugging Face transformers in one line. llama.cpp and mlx make local INT4/INT8 inference trivial. TensorRT-LLM and vLLM support quantized kernels natively. You can quantize any model on the Hub without permission.

Distillation tooling is fragmented. No standard library covers the full pipeline: logit generation, student config design, loss implementation, curriculum scheduling. You’ll write custom code. The ecosystem has recipes (DistilBERT, TinyLlama, Phi distillation) but no turnkey solution. Licensing matters: distilling GPT-4 outputs violates OpenAI’s terms. Distilling Llama-3 is fine.

# Quantization: one-liner in vLLM
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3-8B-Instruct",
    quantization="awq",  # or "gptq", "bitsandbytes"
    dtype="half",
)

# Distillation: you build the pipeline
# 1. Generate teacher logits (sharded, compressed storage)
# 2. Define student architecture
# 3. Implement custom Trainer with distillation loss
# 4. Design curriculum: temperature decay, alpha schedule, data mixing
# 5. Train with FSDP/DeepSpeed
# 6. Evaluate on held-out benchmarks + your task-specific evals

Limits and failure modes

Quantization fails when:

• The model has outlier weights that break symmetric quantization (common in attention outputs)
• You push below INT4 without QAT and careful group-wise quantization
• Activation quantization introduces unbounded error accumulation in long contexts
• The calibration set doesn’t cover your deployment distribution

Distillation fails when:

• The student capacity is too low for the task complexity (capacity gap)
• The teacher’s logits are overconfident, providing weak gradients (temperature helps but doesn’t fix)
• The distillation data lacks diversity — the student memorizes teacher patterns instead of learning generalizable features
• You distill a proprietary model and lose access to the teacher for re-distillation

Both techniques struggle with structured reasoning. Quantization degrades chain-of-thought fidelity at low bit-widths. Distillation rarely teaches the student to reason; it teaches the student to mimic the teacher’s reasoning traces. If the teacher’s reasoning is flawed, the student learns the flaw.

Which to choose

Choose quantization when:

• You need a drop-in replacement for an existing model with minimal engineering investment
• Broad capability preservation matters (general chat, coding, RAG)
• You control the model weights and can re-quantize on updates
• Latency requirements are met by 2-4x speedup
• Your team lacks distributed training infrastructure

Choose distillation when:

• You need 5-10x latency reduction and can invest 6+ engineer-weeks
• The deployment target has hard memory constraints (<2 GB VRAM, mobile/edge)
• You have a narrow, well-defined task distribution (classified ads, support triage, SQL generation)
• You own the teacher model and can re-distill on updates
• You need a model that doesn’t know things outside your domain (compliance, safety)

Choose both when:

• You’ve distilled to a small student and still need 2x more speed
• You’re serving at scale and the combined artifact fits your hardware budget
• You can amortize the distillation cost over millions of daily requests

# Practical decision heuristic
def choose_compression_strategy(
    latency_budget_ms: float,
    vram_budget_gb: float,
    task_breadth: str,  # "narrow" | "broad"
    engineering_weeks: int,
    teacher_access: bool,
) -> str:
    if task_breadth == "broad" or not teacher_access:
        if engineering_weeks < 4:
            return "quantization_only"
        return "quantization_then_evaluate"
    
    # Narrow task, teacher access
    if latency_budget_ms < 20 and vram_budget_gb < 4:
        if engineering_weeks >= 8:
            return "distill_then_quantize"
        return "quantize_teacher_heavily"  # INT4 + speculative decoding
    
    return "quantization_only"

The honest answer: most teams start with quantization, hit a wall on latency or memory, then invest in distillation for their highest-volume endpoints. n4n.ai sees this pattern across inference workloads — quantization gets you to production, distillation keeps you there when traffic scales. Start simple, measure, then complicate.