Topic
Knowledge Distillation
7 posts on knowledge distillation — part of glossary on the n4n AI blog.
What is knowledge distillation in machine learning?
A practitioner's guide to knowledge distillation — how teacher-student training compresses models, when to use it, and where it falls short.
Knowledge distillation vs quantization: the difference
A practitioner's comparison of knowledge distillation vs quantization for LLM compression — covering capabilities, latency, cost, ergonomics, and when to use each.
Knowledge distillation explained: teacher and student
Knowledge distillation explained: train a smaller student model to mimic a larger teacher's output distributions, preserving capability while cutting inference cost.
How Llama 3.2 1B was distilled from larger Llama models
Technical deep-dive into Meta's Llama 3.2 1B distillation process, covering teacher-student architecture, logit matching, and deployment tradeoffs for edge inference.
How Gemma 2 uses knowledge distillation
Deep dive into Gemma 2's knowledge distillation pipeline — teacher-student architecture, loss functions, and practical tradeoffs for deploying smaller models.
How DistilBERT was distilled from BERT
A technical deep-dive into DistilBERT's distillation process from BERT, covering the triple loss function, architectural changes, and practical tradeoffs for engineers.
Distillation vs pruning: two ways to shrink a model
A practitioner's comparison of distillation vs pruning for LLM compression — covering quality, latency, tooling, and when to use each.
More topics in glossary
- Structured Outputs & JSON Mode19
- AI Agents Fundamentals12
- Hallucination in LLMs11
- Sampling Parameters: Top-p, Top-k & Penalties11
- Context Window & Context Length10
- Fine-Tuning Fundamentals9
- Foundation Models: Base vs Instruct vs Chat9
- Model Families & Naming Conventions: GPT-5, Claude, Gemini 3, Llama 4, Mistral, DeepSeek, Qwen, Grok9
- Grounding & Fact-Checking in AI8
- LLM Benchmarks: MMLU, HumanEval, SWE-bench & GPQA8
- Max Tokens, Stop Sequences & Output Truncation8
- Quantization Formats: GGUF, GPTQ, AWQ & INT4/INT88