How Llama 3.2 1B was distilled from larger Llama models

Meta’s Llama 3.2 1B distillation represents one of the most practical applications of knowledge distillation at scale. The 1B and 3B models aren’t just smaller checkpoints — they’re the output of a deliberate teacher-student pipeline that compresses the 11B and 90B vision-language models into packages that run on-device. For engineers targeting mobile or edge deployment, understanding exactly how this distillation works — and where it falls short — determines whether you can trust these models in production.

The distillation thesis

Llama 3.2 1B distillation follows a two-stage recipe: structured pruning followed by logit-based knowledge distillation. The 1B model descends from the 11B vision-language teacher; the 3B model comes from the 90B teacher. This isn’t simple quantization or naive fine-tuning on teacher outputs. The pipeline preserves the teacher’s reasoning traces while stripping parameters that contribute minimally to the final distribution.

The core insight: a 1B parameter model trained from scratch on the same data would underperform significantly. By initializing from a pruned teacher and distilling logits, the student inherits the teacher’s “dark knowledge” — the relative probabilities across the vocabulary that encode reasoning structure, not just the argmax predictions.

Stage 1: Structured pruning as initialization

Before distillation begins, Meta applies structured pruning to the teacher model. This isn’t magnitude-based weight pruning (which destroys structured sparsity patterns). Instead, they remove entire attention heads and MLP blocks based on sensitivity analysis — measuring each component’s contribution to the loss on a calibration set.

# Conceptual pruning sensitivity scoring
def compute_head_importance(model, calibration_data):
    importances = {}
    for layer_idx, layer in enumerate(model.layers):
        for head_idx in range(layer.num_heads):
            # Ablate head and measure loss delta
            with ablate_head(layer, head_idx):
                loss = evaluate(model, calibration_data)
            importances[(layer_idx, head_idx)] = loss - baseline_loss
    return importances

The pruning target: reduce 11B → ~1.3B parameters before distillation. The remaining architecture keeps the same hidden dimension (2048) but cuts layers from 48 to 16 and attention heads from 32 to 16. This preserves the model’s “width” while collapsing depth — a deliberate choice because width correlates more strongly with knowledge capacity than depth.

Why this matters for you: if you’re considering further pruning a 1B model for faster inference, you’re already operating on a depth-collapsed architecture. Aggressive layer dropping will hurt disproportionately compared to pruning a full-depth model.

Stage 2: Logit distillation with temperature scaling

The pruned student then trains on a mixture of ground-truth tokens and teacher logits. The distillation objective combines standard cross-entropy with a KL divergence term matching the teacher’s softened probability distribution:

def distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):
    # Hard target loss (ground truth)
    ce_loss = F.cross_entropy(student_logits, labels)
    
    # Soft target loss (teacher distribution)
    student_soft = F.log_softmax(student_logits / temperature, dim=-1)
    teacher_soft = F.softmax(teacher_logits / temperature, dim=-1)
    kl_loss = F.kl_div(student_soft, teacher_soft, reduction='batchmean') * (temperature ** 2)
    
    return alpha * ce_loss + (1 - alpha) * kl_loss

Temperature (typically 2.0–4.0) softens the teacher distribution, revealing the relative ranking of incorrect tokens. This is where the “dark knowledge” lives: the teacher assigns 0.001% to “apple” and 0.0008% to “orange” for a fruit context — both near-zero, but the ratio encodes semantic structure the student learns to replicate.

Meta’s training run uses ~9T tokens for the 1B model — roughly the same token budget as the original 11B pre-training. The distillation phase consumes the bulk of this budget. The student sees each token twice: once with the teacher’s logits (distillation), once with standard labels (ground-truth anchoring). This prevents catastrophic forgetting of basic language modeling capability.

Data composition and curriculum

The distillation dataset mirrors the teacher’s pre-training mix: CommonCrawl, code, multilingual text, and synthetic reasoning traces. But there’s a critical addition — the teacher generates its own reasoning traces for complex prompts, and the student distills on those traces. This is effectively self-distillation at scale: the 11B model produces chain-of-thought for math and coding problems, and the 1B student learns to mimic the reasoning pattern, not just the final answer.

{
  "prompt": "Solve: 3x + 7 = 22",
  "teacher_trace": "Subtract 7 from both sides: 3x = 15. Divide by 3: x = 5.",
  "teacher_logits": [...],  // Full vocab distribution at each step
  "student_target": "x = 5"
}

This trace distillation is why the 1B model punches above its weight on GSM8K and HumanEval relative to from-scratch 1B baselines. It’s not memorizing answers — it’s learning the teacher’s algorithmic reasoning style.

Quantization-aware distillation

A practical detail buried in the release notes: distillation targets INT4 quantization from day one. The student trains with fake quantization nodes (straight-through estimator) so the logits it learns to match are already the quantized teacher’s logits. This avoids the post-training quantization gap that plagues most distilled models.

# Fake quantization during distillation training
class QuantizedLinear(nn.Module):
    def __init__(self, weight, bits=4):
        super().__init__()
        self.weight = nn.Parameter(weight)
        self.scale = nn.Parameter(torch.tensor(1.0))
        self.zero_point = nn.Parameter(torch.tensor(0))
        self.bits = bits
    
    def forward(self, x):
        # STE quantization
        w_q = fake_quantize(self.weight, self.scale, self.zero_point, self.bits)
        return F.linear(x, w_q)

If you’re deploying Llama 3.2 1B via n4n.ai or any gateway that serves quantized weights, this matters: the model you’re calling was distilled to match quantized teacher behavior, not FP16 teacher behavior. The distillation objective already accounts for quantization noise.

Where the distillation breaks down

The 1B model fails predictably in three regimes:

1. Long-context reasoning. The teacher’s 128K context window compresses to 128K in the student architecturally, but the distilled model loses the ability to use it effectively beyond ~16K tokens. The pruning removed layers that implement the “memory retrieval” attention patterns. You’ll see attention entropy collapse on long contexts — the model attends uniformly instead of selectively.

2. Multilingual low-resource languages. The teacher’s multilingual capacity relies on dedicated capacity per language family. At 1B parameters, the student must share capacity across all languages. Distillation preserves high-resource language performance (English, Chinese, Spanish) but low-resource languages degrade to near-random. The teacher’s logits for low-resource tokens are noisy; the student amplifies that noise.

3. Compositional generalization. The student mimics the teacher’s reasoning style but not its depth. On multi-hop reasoning requiring 5+ steps, the 1B model hallucinates intermediate conclusions. The teacher’s logits encode uncertainty at each step; the student’s distilled logits are overconfident because temperature scaling during distillation suppresses the teacher’s calibrated uncertainty.

Benchmark reality check

The distillation gain over from-scratch training is real and consistent: ~15-25 points absolute across benchmarks. But the gap to the teacher remains 15-35 points. Distillation recovers knowledge density but not reasoning capacity. The 1B model knows what the teacher knows, but lacks the working memory to manipulate that knowledge through long chains.

Deployment implications

For edge deployment, the 1B model at INT4 quantizes to ~700MB — small enough for on-device inference on modern phones (Neural Engine, NPU, or GPU). Latency: ~30-50ms/token on iPhone 15 Pro, ~80-120ms/token on Snapdragon 8 Gen 3. This is the distillation’s real victory: a model that fits in mobile memory budgets while retaining usable reasoning.

But you need to architect around the failure modes:

# Practical deployment guardrails for Llama 3.2 1B
class EdgeLlamaGuardrails:
    def __init__(self, model, tokenizer):
        self.model = model
        self.tokenizer = tokenizer
        self.max_reasoning_steps = 3  # Hard limit
        self.context_window = 8192    # Effective limit, not 128K
    
    def generate(self, prompt, **kwargs):
        # Truncate context aggressively
        tokens = self.tokenizer.encode(prompt)
        if len(tokens) > self.context_window:
            tokens = tokens[-self.context_window:]
        
        # Inject step-limiter for reasoning tasks
        if self._is_reasoning_task(prompt):
            kwargs['stop_sequences'] = kwargs.get('stop_sequences', []) + ['\n\n']
            kwargs['max_new_tokens'] = min(kwargs.get('max_new_tokens', 512), 256)
        
        return self.model.generate(tokens, **kwargs)

The decisive takeaway

Llama 3.2 1B distillation proves that structured pruning + logit distillation + quantization-aware training can compress a 11B vision-language model to 1B parameters with ~80% knowledge retention on standard benchmarks. But the compression is lossy in specific, predictable ways: long-context retrieval, low-resource multilingual, and deep compositional reasoning degrade disproportionately.

If your use case fits in 8K context, targets high-resource languages, and requires ≤3 reasoning steps — the 1B model is a remarkable engineering artifact. Deploy it confidently. If you need any of the three failure modes, you’re not saving compute by using the distilled model; you’re buying latency at the cost of correctness. Route those requests to the 11B or 90B teacher instead.