Benchmarking LLMs: perplexity vs MMLU vs human eval

You’re shipping an LLM feature and need to know if your model change actually improved things. Perplexity drops on your validation set. MMLU ticks up two points. Your PM says “but does it feel better?” These three signals often disagree. Understanding why they disagree — and which to trust for your specific use case — separates teams that ship reliable AI from teams that chase metrics off a cliff.

What each metric actually measures

Perplexity measures how surprised a model is by the next token. Formally, it’s the exponential of average negative log-likelihood: exp(-1/N Σ log P(token_i | context)). Lower is better. It’s a pure compression metric — how well does the model predict the training distribution?

MMLU (Massive Multitask Language Understanding) tests multiple-choice knowledge across 57 subjects — STEM, humanities, social sciences. It’s a capability benchmark: can the model recall facts, reason through problems, and apply knowledge? Score is percentage correct.

Human evaluation puts real people in the loop. They rank outputs, score on rubrics, or choose between model A and model B. It measures whatever your evaluators care about: helpfulness, safety, tone, instruction following, creative quality.

These measure fundamentally different things. Perplexity measures modeling fidelity to a distribution. MMLU measures knowledge retrieval and reasoning on curated tasks. Human eval measures utility to actual users. They correlate, but loosely.

When perplexity lies

Perplexity is cheap, automatic, and differentiable — you can optimize it directly. That’s why it’s the default training objective. But it has sharp edges.

First, perplexity is domain-bound. A model trained on code will have terrible perplexity on legal contracts, even if it’s excellent at both. Comparing perplexity across different tokenizers is meaningless without normalization. Comparing across different validation sets is equally meaningless.

Second, perplexity doesn’t capture useful behavior. A model can achieve low perplexity by being verbose, hedging, or repeating common patterns — behaviors users hate. The classic example: a chat model that says “I’m not sure, but…” before every answer. Perplexity loves it. Users don’t.

Third, perplexity saturates. Once you’re below ~2.0 on your domain, further drops rarely translate to perceptible quality gains. You’re optimizing noise.

# Perplexity calculation — simple but context-dependent
import math
import torch

def perplexity(log_probs: torch.Tensor) -> float:
    """log_probs shape: (batch, seq_len) — already log P(token|context)"""
    return math.exp(-log_probs.mean().item())

# Critical: this only makes sense on the SAME tokenizer, SAME domain
# as your training data. Cross-domain comparison = garbage.

Use perplexity for: training monitoring, early stopping, comparing checkpoints of the same architecture on the same data. Don’t use it for: model selection across architectures, release decisions, or claiming “model A is better than model B.”

What MMLU actually tells you

MMLU became the de facto standard because it’s broad, reproducible, and correlates reasonably with downstream performance. But it has structural flaws engineers should know.

The test set leaked into training data for many models. Scores on public MMLU are inflated — sometimes severely. The “clean” MMLU variants (MMLU-Pro, MMLU-Redux) exist partly because of this. If you’re evaluating a model you didn’t train, assume contamination unless proven otherwise.

MMLU is multiple choice. Real use cases are open-ended. A model can ace MMLU by recognizing answer patterns without genuine reasoning. Chain-of-thought prompting helps, but the benchmark format still constrains what’s measured.

Subject coverage is broad but shallow. Five questions per subject means high variance. A 2-point swing can be noise. Confidence intervals matter — most leaderboards don’t show them.

{
  "mmlu_caveats": [
    "Test set contamination in most public models",
    "Multiple-choice format ≠ open-ended capability",
    "High variance: ±2-3 points typical per subject",
    "English-centric, Western knowledge bias",
    "No measure of instruction following or safety"
  ]
}

Use MMLU for: rough capability tiering (is this a 60% model or an 80% model?), comparing base models before fine-tuning, tracking the field’s progress. Don’t use it for: evaluating your fine-tuned model on your task, measuring instruction following, or replacing human eval.

Human evaluation: the only metric that matters — if you do it right

Human eval is expensive, slow, and noisy. It’s also the only metric that correlates with your product outcomes. But most teams do it poorly.

The standard pairwise comparison (A vs B) has known biases: position bias (first output wins), length bias (longer wins), style bias (confident tone wins). You need counterbalancing, multiple annotators per comparison, and statistical testing.

# Minimal pairwise eval with counterbalancing
import random
from dataclasses import dataclass
from typing import Literal

@dataclass
class Comparison:
    prompt: str
    output_a: str
    output_b: str
    winner: Literal["A", "B", "TIE"]
    annotator_id: str

def run_pairwise_eval(prompts, model_a, model_b, annotators, n_per_annotator=20):
    comparisons = []
    for prompt in random.sample(prompts, n_per_annotator * len(annotators)):
        a_out = model_a.generate(prompt)
        b_out = model_b.generate(prompt)
        # Counterbalance position
        if random.random() < 0.5:
            comparisons.append(Comparison(prompt, a_out, b_out, None, ""))
        else:
            comparisons.append(Comparison(prompt, b_out, a_out, None, ""))
    # Distribute to annotators with clear rubric
    return comparisons

Rubric design is everything. “Which is better?” produces noise. “Which follows instructions more faithfully?” produces signal. Define your dimensions explicitly: instruction following, factual accuracy, tone, safety, conciseness. Score each dimension separately.

Inter-annotator agreement (Cohen’s kappa, Krippendorff’s alpha) should be reported. If kappa < 0.6, your rubric is ambiguous or your annotators are untrained. Fix the rubric before collecting more data.

Use human eval for: release decisions, comparing models on your tasks, measuring qualities no automatic metric captures. Don’t use it for: rapid iteration (too slow), hyperparameter search (too expensive), or as a single-number leaderboard (multidimensional).

Comparison across dimensions

Combining them: a practical evaluation stack

No single metric suffices. The teams that ship confidently run a tiered evaluation pipeline:

Tier 1 — Automatic, every commit: Perplexity on held-out validation sets (per domain), plus a small suite of automatic evals — exact match, ROUGE, BERTScore, or LLM-as-judge on your task distribution. Fast, cheap, catches regressions.

Tier 2 — Weekly/per release candidate: MMLU (clean variant) + your task-specific benchmarks (coding, summarization, RAG, whatever your product does). This is your capability regression test.

Tier 3 — Per major release: Human evaluation on a representative sample of real user prompts. Minimum 200 comparisons, 3 annotators each, counterbalanced, with dimension-specific rubrics. This is your release gate.

# Example eval pipeline config
tier1:
  frequency: "every_commit"
  metrics:
    - perplexity: {domains: ["code", "chat", "docs"]}
    - llm_judge: {rubric: "instruction_following", sample_size: 100}
  threshold: "no_regression_p95"

tier2:
  frequency: "weekly"
  benchmarks:
    - mmlu_pro: {subset: "stem"}
    - humaneval: {pass_at_k: 1}
    - custom_rag: {dataset: "production_sample_500"}
  threshold: "mmlu_pro >= 75, humaneval_pass1 >= 60"

tier3:
  frequency: "major_release"
  human_eval:
    sample_size: 300
    annotators_per_item: 3
    dimensions: ["instruction_following", "accuracy", "tone", "safety"]
    counterbalance: true
  threshold: "win_rate_vs_production >= 55% on all dimensions"

Which to choose — by use case

Training a base model from scratch: Perplexity is your north star. Track it on multiple validation domains. MMLU is a periodic checkpoint. Human eval is irrelevant until you have a chat model.

Fine-tuning for a specific task: Perplexity on your task data (monitor only). Task-specific automatic metrics (exact match, pass@k, ROUGE). Human eval on your task — this is your release gate. MMLU is a sanity check that you didn’t destroy general capability.

Choosing between API models for production: MMLU gives you a rough tier. But run your own Tier 2 benchmarks on your actual prompts. Then Tier 3 human eval on the top 2-3 candidates. The model that wins MMLU often loses your task-specific eval.

Debugging a quality regression: Perplexity tells you if the model changed. Automatic evals tell you what broke. Human eval confirms the user impact. Start with perplexity — if it’s stable, the regression is in your prompt, retrieval, or post-processing, not the model.

Proving ROI to leadership: Human eval tied to product metrics. “Model B wins 62% on instruction following, and in A/B test that translated to 8% higher task completion.” Perplexity and MMLU don’t speak business language.

The trap to avoid

The most common mistake: treating these as interchangeable “quality scores.” They’re not. A 5% perplexity drop, a 3-point MMLU gain, and a 55% human win rate are three different statements about three different things.

The second most common mistake: skipping Tier 3 because it’s slow. If you’re shipping user-facing LLM features without human evaluation on your tasks, you’re guessing. Sometimes guessing works. Often it doesn’t. The cost of a bad release exceeds the cost of a proper eval by orders of magnitude.

Build the pipeline. Automate Tiers 1 and 2. Invest in Tier 3 rigor. That’s how you ship models that actually work.