How to read an LLM model card's benchmark table

Model cards are the spec sheets of the LLM world, but their benchmark tables read like alphabet soup: MMLU 82.3, HumanEval 91.2, GPQA 48.7, SWE-bench 14.5. If you’ve ever stared at these numbers wondering which ones actually predict how a model will perform on your task, you’re not alone. This guide walks through how to read llm benchmark table data systematically, so you can stop guessing and start selecting models with evidence.

Step 1: Identify the benchmark suite and version

Every benchmark table should list the exact benchmark name and version. MMLU alone has multiple variants — 5-shot, 0-shot, STEM subset, humanities subset — and scores are not comparable across them. The same applies to HumanEval (pass@1 vs pass@10), GPQA (diamond vs main vs extended), and SWE-bench (verified vs full).

Look for a footnote or methodology link. If the card says “MMLU: 82.3” without qualification, assume 5-shot on the full test set, but verify. A well-documented card will reference the evaluation harness: lm-evaluation-harness commit hash, inspect_ai version, or the provider’s internal eval script.

{
  "benchmark": "MMLU",
  "version": "5-shot",
  "harness": "lm-evaluation-harness@0.4.3",
  "commit": "a1b2c3d",
  "date": "2024-11-15"
}

If this metadata is missing, treat the number as directional at best.

Step 2: Map benchmarks to your actual workload

Benchmarks measure specific capabilities. Match them to what your application does:

Don’t optimize for MMLU if you’re building a coding agent. A model with MMLU 85 and SWE-bench 12 will frustrate you more than one with MMLU 78 and SWE-bench 28.

Step 3: Check the evaluation protocol details

Two models scoring “82 on MMLU” can behave differently because of evaluation choices. Scrutinize these factors:

Prompt format: Some evals use the model’s chat template, others use a raw completion prompt. Chat-tuned models often score lower on raw completion benchmarks because they expect conversation structure.

Few-shot examples: MMLU 5-shot means five examples per question. The specific examples chosen (and their order) can swing scores by 2-3 points. Standardized example sets exist — check if the card references them.

Temperature and sampling: Most academic benchmarks run at temperature 0 (greedy) or with a fixed seed. Production traffic rarely uses temperature 0. If the card doesn’t state the sampling config, assume greedy decoding.

Token limits: Some evaluations truncate context; others use the model’s full window. A model with 128k context evaluated at 4k may show inflated scores on long-context tasks.

Post-processing: Code benchmarks often apply sanitization (removing markdown fences, fixing indentation) before execution. If the card doesn’t describe this, the pass@1 number may not reflect raw model output.

# Example: How lm-evaluation-harness runs MMLU 5-shot
from lm_eval import evaluator
from lm_eval.models.huggingface import HFLM

model = HFLM(pretrained="meta-llama/Meta-Llama-3.1-70B-Instruct")
results = evaluator.simple_evaluate(
    model=model,
    tasks=["mmlu"],
    num_fewshot=5,
    batch_size=8,
    limit=None,  # full test set
)
print(results["results"]["mmlu"]["acc,none"])

Run this yourself against a candidate model to verify the card’s claim.

Step 4: Distinguish between base and instruct scores

Base models (pre-trained only) and instruct models (post-trained for chat) are different products. A base model scoring 75 MMLU may become 82 after instruction tuning — or drop to 70 if the tuning degrades raw reasoning.

Model cards sometimes blend these. Look for labels like “Llama-3.1-70B” (base) vs “Llama-3.1-70B-Instruct”. If the card lists both, compare the delta. A large drop on reasoning benchmarks after instruction tuning can signal over-optimization for chat style at the expense of capability.

Step 5: Read the confidence intervals and sample sizes

A single-number score hides variance. Proper benchmark reporting includes:

• Standard error or 95% confidence interval: MMLU has ~14k questions. A 0.5% standard error means the true score is likely within ±1 point.
• Number of samples evaluated: Some providers evaluate on a subset (e.g., 1000 random MMLU questions) to save compute. This widens the confidence interval significantly.
• Multiple seeds: Code benchmarks should report pass@k across multiple seeds (typically n=20-200). Single-seed results are noisy.

If the card shows “HumanEval: 91.2” without “pass@1, n=200, temp=0”, ask for the raw logs. n4n.ai surfaces per-model evaluation metadata when available so you can see the exact harness configuration behind each number.

Step 6: Spot contamination and data leakage red flags

Benchmark contamination — where test data appears in training — inflates scores without improving real capability. Warning signs:

• Suspiciously high scores on new benchmarks: If a model beats the previous SOTA on GPQA by 15 points but only 2 points on MMLU, investigate.
• Perfect scores on known-contaminated subsets: Some MMLU subsets (e.g., “college_mathematics”) have known leakage. Check the contamination analysis for the benchmark version used.
• No holdout evaluation: Reputable labs evaluate on held-out test sets not seen during development. If the card only reports public benchmark scores, assume some contamination.

When in doubt, run a quick contamination check: sample 20 questions from the benchmark, search the model’s training data sources (Common Crawl, Wikipedia, GitHub) for exact matches. This takes 10 minutes and catches obvious leakage.

Step 7: Compare apples to apples with a control model

Never evaluate a model in isolation. Pick a control model you know well — one you’ve run in production — and evaluate both on the same harness, same hardware, same prompt format.

# Run identical eval on two models for direct comparison
lm_eval --model hf \
  --model_args pretrained=meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tasks mmlu,humaneval,gpqa \
  --num_fewshot 5 \
  --batch_size 8 \
  --output_path ./eval_results/llama31_70b_instruct.json

lm_eval --model hf \
  --model_args pretrained=Qwen/Qwen2.5-72B-Instruct \
  --tasks mmlu,humaneval,gpqa \
  --num_fewshot 5 \
  --batch_size 8 \
  --output_path ./eval_results/qwen25_72b_instruct.json

Compare the JSON outputs. Differences in prompt template handling, tokenizer behavior, or generation config will show up here — and they often explain why a model “feels” better or worse than its benchmark table suggests.

Step 8: Weight benchmarks by your cost and latency constraints

A model scoring 5 points higher on SWE-bench may cost 3x more per token and run 2x slower. Build a simple scoring function:

def model_score(benchmarks, pricing, latency_p50, weights):
    """
    benchmarks: dict of {benchmark_name: score_0_to_100}
    pricing: dict with 'input_per_1m', 'output_per_1m' (USD)
    latency_p50: median time to first token (ms)
    weights: dict of {benchmark_name: importance_weight}
    """
    # Normalize benchmarks to 0-1
    norm_bench = {k: v / 100 for k, v in benchmarks.items()}
    
    # Weighted capability score
    capability = sum(norm_bench[k] * weights.get(k, 0) for k in norm_bench)
    
    # Cost penalty (log scale, tuned to your budget)
    cost_per_1k = (pricing['input_per_1m'] + pricing['output_per_1m']) / 1000
    cost_penalty = 0.1 * (cost_per_1k ** 0.5)
    
    # Latency penalty
    latency_penalty = 0.05 * (latency_p50 / 1000) ** 0.5
    
    return capability - cost_penalty - latency_penalty

# Example weights for a coding agent
weights = {
    "swe_bench_verified": 0.5,
    "humaneval_pass_at_1": 0.3,
    "mmlu": 0.1,
    "gpqa": 0.1,
}

Adjust weights to your priorities. This forces explicit trade-offs instead of chasing the highest MMLU.

Step 9: Verify with a production-like eval set

Benchmarks are proxies. The only score that matters is performance on your actual task distribution. Build a small (50-200 example) eval set from real or synthetic data that mirrors your production traffic.

# Minimal eval harness for your task
import json
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

def evaluate_model(model_id, eval_cases, judge_prompt):
    results = []
    for case in eval_cases:
        response = client.chat.completions.create(
            model=model_id,
            messages=[{"role": "user", "content": case["input"]}],
            temperature=0.7,
        )
        output = response.choices[0].message.content
        
        # LLM-as-judge for open-ended tasks
        judge = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": judge_prompt},
                {"role": "user", "content": f"Input: {case['input']}\nOutput: {output}\nExpected: {case.get('expected', 'N/A')}"}
            ],
            temperature=0,
        )
        score = float(judge.choices[0].message.content.strip())
        results.append({"input": case["input"], "output": output, "score": score})
    
    avg_score = sum(r["score"] for r in results) / len(results)
    return avg_score, results

# Run on your shortlist
for model in ["meta-llama/Meta-Llama-3.1-70B-Instruct", "Qwen/Qwen2.5-72B-Instruct"]:
    score, details = evaluate_model(model, my_eval_cases, my_judge_prompt)
    print(f"{model}: {score:.2f}")

This catches failure modes benchmarks miss: tone drift, format violations, domain-specific hallucinations, latency spikes under load.

Step 10: Document your decision and set a re-evaluation trigger

Record what you chose, why, and when you’ll revisit. Model capabilities shift fast — new quantization methods, fine-tunes, and provider optimizations change the landscape monthly.

## Model Selection Record: Coding Agent v2.1

**Selected**: Qwen2.5-72B-Instruct (via n4n.ai)
**Date**: 2025-01-15
**Benchmark scores (lm-eval-harness@0.4.3, 5-shot, temp=0)**:
- SWE-bench Verified: 28.4% (control: Llama-3.1-70B-Instruct 22.1%)
- HumanEval pass@1: 91.2% (control: 88.7%)
- MMLU: 84.1% (control: 82.3%)

**Production eval (200 cases, LLM-as-judge)**: 4.2/5.0 vs control 3.8/5.0
**Cost**: $0.35/1M in, $0.40/1M out (vs $0.90/$0.90 for control)
**Latency p50**: 420ms TTFT (vs 380ms control)

**Re-evaluation trigger**: 
- New SWE-bench SOTA exceeds 35% on verified subset
- Quarterly (next: 2025-04-15)
- If production error rate > 2% on coding tasks

This discipline prevents vendor lock-in and ensures you’re not running a six-month-old model because “it worked last time.”

Verification checklist

Before you commit to a model based on its card:

• Benchmark names include version and harness commit
• At least three benchmarks map directly to your workload
• Evaluation protocol (prompt format, temperature, few-shot) is documented
• Base vs instruct distinction is clear
• Confidence intervals or sample sizes are reported
• No obvious contamination red flags
• You’ve run a head-to-head comparison against a control model on the same harness
• You’ve scored candidates on a cost/latency-adjusted utility function
• You’ve validated on a production-like eval set (minimum 50 cases)
• Decision is documented with a re-evaluation trigger

If any box is unchecked, you don’t have enough signal — you have marketing. Run the evals.