What is MMLU? The multitask benchmark explained

MMLU (Massive Multitask Language Understanding) is a benchmark that evaluates language models across 57 subjects spanning STEM, humanities, social sciences, and professional domains. It consists of roughly 16,000 multiple-choice questions designed to test both world knowledge and problem-solving ability in a zero-shot or few-shot setting. If you’re asking what is mmlu because you need to compare models for a production workload, this is the reference most vendors cite — but it has sharp edges you should understand before relying on it.

How the benchmark works

Each MMLU subject contains a few hundred questions, each with four answer choices (A–D). The standard evaluation protocol feeds the model a prompt containing the question, the four options, and optionally a few exemplars from the same subject (5-shot is the original paper’s default). The model outputs a single letter, which is scored as correct or incorrect. Accuracy is reported per subject and macro-averaged across all 57 subjects to produce the headline number you see in model cards.

# Minimal zero-shot prompt format
prompt = """The following are multiple choice questions (with answers) about {subject}.

Question: {question}
A. {option_a}
B. {option_b}
C. {option_c}
D. {option_d}
Answer:"""

The 57 subjects group into four broad categories:

Category	Example subjects	Question count (approx.)
STEM	abstract_algebra, college_physics, computer_security, electrical_engineering	4,500
Humanities	formal_logic, high_school_european_history, philosophy, world_religions	3,800
Social sciences	econometrics, high_school_psychology, professional_law, public_relations	3,200
Other (professional/vocational)	business_ethics, clinical_knowledge, human_sexuality, marketing	4,500

The original paper (Hendrycks et al., 2021) reports human expert performance at ~89.8% average accuracy, while random chance is 25%. Current frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) sit in the 85–90% range on the full suite.

Why engineers care about MMLU

For model selection, MMLU serves as a coarse but useful filter. If you’re building a legal research assistant, you care about professional_law and international_law subscores more than the aggregate. If you’re routing coding tasks, computer_security and high_school_computer_science correlate better with your use case than the overall number.

// Example: extracting subscores from a provider's eval dump
{
  "model": "gpt-4o-2024-08-06",
  "mmlu": {
    "overall": 0.887,
    "subcategories": {
      "professional_law": 0.921,
      "computer_security": 0.843,
      "clinical_knowledge": 0.867,
      "abstract_algebra": 0.712
    }
  }
}

MMLU also enables apples-to-apples comparison across open and closed models. When a new Llama or Qwen release drops, the first thing the community does is run MMLU (usually 5-shot) so you can slot it into your routing logic without running a full eval suite yourself.

A practical pattern: use MMLU subscores to build a routing table that sends legal queries to the model with the highest professional_law score, coding tasks to the one leading on computer_security, and so on. This is exactly the kind of per-task routing directive that an inference gateway can honor automatically — n4n.ai, for instance, lets you specify routing preferences that map to these capability dimensions without hardcoding model names in your application code.

Concrete example: running MMLU yourself

You don’t need a GPU cluster to reproduce a subset. The lm-evaluation-harness from EleutherAI is the de facto standard runner.

# Install
pip install lm-eval

# Run 5-shot on a single subject against a local HF model
lm_eval --model hf \
  --model_args pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct \
  --tasks mmlu_professional_law \
  --num_fewshot 5 \
  --batch_size 4 \
  --output_path ./results

For API models, the same harness supports OpenAI-compatible endpoints:

lm_eval --model openai-completions \
  --model_args model=gpt-4o,base_url=https://api.openai.com/v1 \
  --tasks mmlu \
  --num_fewshot 5

The harness handles prompt formatting, answer extraction (it parses “Answer: B” or just “B”), and macro-averaging. Expect a full 57-subject run on a 70B model to take 2–4 hours on 4×A100; an 8B model finishes in ~45 minutes.

If you only need a quick signal, run the mmlu_stem or mmlu_humanities aggregate tasks instead of all 57 individually — they’re predefined in the harness and finish in a fraction of the time.

Common misconceptions

“Higher MMLU means better for my use case”

Not necessarily. MMLU measures multiple-choice knowledge retrieval and reasoning on academic-style questions. It does not measure:

Instruction following fidelity (will it actually output valid JSON when asked?)
Long-context retrieval accuracy
Tool use or function calling correctness
Latency, cost, or rate limit behavior
Safety alignment for your specific policy

A model with 86% MMLU that follows your system prompt reliably may outperform an 89% model that ignores formatting constraints. Always run a targeted eval on your actual task distribution.

“5-Shot is the only valid setting”

The original paper used 5-shot, but zero-shot and 3-shot are common in practice. Zero-shot is harder but closer to how you’ll actually prompt the model in production (few-shot exemplars consume context window and increase latency). Report which setting you used — numbers are not comparable across shot counts.

# Zero-shot prompt differs only by omitting exemplars
zero_shot_prompt = """Question: {question}
A. {option_a}
B. {option_b}
C. {option_c}
D. {option_d}
Answer:"""

“MMLU is contamination-free”

It isn’t. The test set has been public since 2021. Every major training run since then has likely seen some portion of it, either directly or via synthetic data generated from it. Treat MMLU as a contaminated benchmark — useful for relative comparison, not absolute capability measurement. For cleaner signal, look at held-out evals like GPQA (graduate-level science) or MMLU-Pro (a harder, less contaminated variant released in 2024).

“The macro average is the right aggregate”

Macro-averaging weights each subject equally regardless of question count. This means high_school_physics (≈200 questions) contributes the same as professional_law (≈150 questions). If your workload is heavily skewed toward one domain, compute a weighted average matching your traffic distribution instead of trusting the headline number.

# Weighted by your actual query distribution
my_weights = {
    "professional_law": 0.35,
    "computer_security": 0.25,
    "clinical_knowledge": 0.20,
    "business_ethics": 0.10,
    "other": 0.10
}
weighted_score = sum(score * my_weights.get(subj, 0.10/47) for subj, score in subscores.items())

What to do next

Run a subset relevant to your domain before committing to a model. The harness makes this trivial.
Track subscores, not just the aggregate. Build a routing table keyed by capability dimension.
Supplement with task-specific evals — MMLU is necessary but not sufficient for production confidence.
Re-evaluate on model updates. Providers silently swap model versions; a 0.5% MMLU drop can correspond to a meaningful regression on your task.

MMLU is the lingua franca of model comparison for a reason: it’s broad, standardized, and runnable on modest hardware. But like any benchmark, it measures what it measures — not what you need. Use it as a filter, not a verdict.