Common language model evaluation metrics, explained

If you’ve ever stared at a model card claiming “state-of-the-art performance” and wondered what that actually means for your production workload, you’re not alone. Language model evaluation metrics are the vocabulary we use to translate model behavior into engineering decisions. Some measure statistical fluency, others measure task correctness, and a growing category measures alignment with human preferences. This guide walks through the metrics you’ll encounter in papers, leaderboards, and production monitoring dashboards — with the context you need to pick the right ones for your use case.

1. Perplexity

Perplexity is the foundational metric for autoregressive language models. It measures how “surprised” a model is by a test sequence, expressed as the exponentiated average negative log-likelihood per token. Lower is better; a perplexity of 1 means the model assigns probability 1 to every token in the test set.

import math
import torch

def perplexity(log_probs: torch.Tensor, mask: torch.Tensor = None) -> float:
    """
    log_probs: [batch, seq_len] log probabilities of target tokens
    mask: optional [batch, seq_len] boolean mask (1 = valid token)
    """
    if mask is None:
        mask = torch.ones_like(log_probs, dtype=torch.bool)
    nll = -(log_probs * mask).sum() / mask.sum()
    return math.exp(nll.item())

Perplexity correlates with downstream task performance — up to a point. It’s sensitive to tokenization differences (a model with a larger vocabulary will typically show lower perplexity on the same text) and doesn’t capture factual correctness, reasoning, or instruction following. Use it for: comparing base model checkpoints during pretraining, detecting distribution shift in your data, or as a sanity check before running expensive evaluations. Don’t use it as your only metric for chat or RAG systems.

2. Accuracy and exact match

For classification, multiple-choice, and closed-form QA tasks, accuracy is the fraction of predictions that exactly match the ground truth. Exact match (EM) is the stricter variant used in extractive QA (e.g., SQuAD): the predicted answer span must match the gold span character-for-character.

def exact_match(pred: str, gold: str, normalize: bool = True) -> bool:
    if normalize:
        pred = pred.strip().lower()
        gold = gold.strip().lower()
    return pred == gold

def accuracy(predictions: list[str], references: list[str]) -> float:
    return sum(exact_match(p, g) for p, g in zip(predictions, references)) / len(predictions)

Accuracy is interpretable and cheap to compute. Its weakness: it treats “Paris” and “London” as equally wrong when the answer is “Berlin,” and it gives zero credit for partially correct responses. For generative tasks where multiple valid answers exist, accuracy alone is misleading. Pair it with softer metrics or human evaluation.

3. F1 score (token-level and span-level)

F1 balances precision and recall at the token level. In extractive QA, it measures overlap between predicted and gold answer spans after tokenization. In summarization or translation, it can be computed over n-grams (see ROUGE below). Token-level F1 is more forgiving than exact match and correlates better with human judgment on span prediction tasks.

from collections import Counter

def token_f1(pred: str, gold: str, tokenizer) -> float:
    pred_tokens = tokenizer(pred)
    gold_tokens = tokenizer(gold)
    common = Counter(pred_tokens) & Counter(gold_tokens)
    num_same = sum(common.values())
    if num_same == 0:
        return 0.0
    precision = num_same / len(pred_tokens)
    recall = num_same / len(gold_tokens)
    return 2 * precision * recall / (precision + recall)

Choose a tokenizer consistent with your evaluation benchmark (often whitespace or a specific subword tokenizer). F1 degrades gracefully with partial overlap, making it the default for SQuAD-style leaderboards. It still ignores semantic equivalence — “USA” vs “United States” scores zero overlap.

4. BLEU and ROUGE

BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are n-gram overlap metrics from machine translation and summarization. BLEU emphasizes precision (penalizing hallucinated n-grams); ROUGE-L emphasizes recall via longest common subsequence. Both compute geometric means of n-gram precisions (BLEU) or F1 scores (ROUGE) with a brevity penalty.

# Using sacrebleu for reproducible BLEU
import sacrebleu

def compute_bleu(predictions: list[str], references: list[list[str]]) -> float:
    """references is list of list: each prediction can have multiple references"""
    bleu = sacrebleu.corpus_bleu(predictions, references, tokenize='13a')
    return bleu.score  # 0-100 scale

# Using rouge-score for ROUGE-L
from rouge_score import rouge_scorer

def compute_rouge_l(predictions: list[str], references: list[str]) -> float:
    scorer = rouge_scorer.RougeScorer(['rougeL'], use_stemmer=True)
    scores = [scorer.score(ref, pred)['rougeL'].fmeasure
              for pred, ref in zip(predictions, references)]
    return sum(scores) / len(scores)

These metrics are fast, deterministic, and widely reported. They correlate poorly with human judgment on open-ended generation, creative writing, or factual QA. Use them for: translation, summarization, and constrained generation where n-gram fidelity matters. Avoid them as primary metrics for chat, coding, or reasoning tasks.

5. BERTScore and semantic similarity metrics

BERTScore computes token-level similarity using contextual embeddings from a pretrained transformer (typically RoBERTa or DeBERTa). It matches predicted and reference tokens via greedy cosine similarity, then reports precision, recall, and F1. Unlike n-gram metrics, it captures paraphrase equivalence (“car” ≈ “automobile”).

from bert_score import score as bert_score

def compute_bertscore(predictions: list[str], references: list[str], lang: str = 'en') -> dict:
    P, R, F1 = bert_score(predictions, references, lang=lang, verbose=False)
    return {
        'precision': P.mean().item(),
        'recall': R.mean().item(),
        'f1': F1.mean().item()
    }

BERTScore correlates better with human judgment on semantic adequacy than BLEU/ROUGE. It’s slower (requires a forward pass through an encoder) and introduces its own model dependency — scores shift if you change the backing encoder. Use it when: evaluating paraphrase generation, checking factual consistency in summarization, or as a supplement to n-gram metrics. Don’t treat it as ground truth; it can reward fluent but hallucinated text.

6. LLM-as-judge (pairwise and pointwise)

Using a strong LLM to evaluate another model’s outputs has become the de facto standard for open-ended tasks. Two common protocols: pairwise (model A vs model B, judge picks winner or tie) and pointwise (judge scores a single output on a rubric, e.g., 1-5 on helpfulness, correctness, style).

# Minimal pairwise judge prompt template
PAIRWISE_PROMPT = """You are an impartial evaluator. Compare two responses to the same prompt.

Prompt: {prompt}

Response A: {response_a}

Response B: {response_b}

Which response is better? Consider: instruction following, correctness, clarity, and safety.
Answer with exactly one word: "A", "B", or "Tie"."""

# Pointwise rubric example
POINTWISE_PROMPT = """Rate the following response on a scale of 1-5 for each criterion.

Prompt: {prompt}
Response: {response}

Criteria:
- Instruction following (1-5)
- Factual correctness (1-5)
- Clarity and structure (1-5)
- Safety (1-5)

Output JSON: {{"instruction_following": int, "factual_correctness": int, "clarity": int, "safety": int}}"""

LLM-as-judge scales to thousands of samples and captures nuance that n-gram and embedding metrics miss. Caveats: judge models have their own biases (verbosity bias, self-preference, position bias in pairwise), and results vary with prompt engineering. Mitigate by: randomizing order in pairwise, using multiple judges, calibrating against human annotations on a subset, and reporting confidence intervals. This is currently the best available proxy for human preference at scale.

7. Major benchmark suites (MMLU, GSM8K, HumanEval, BBH, etc.)

Leaderboards aggregate multiple tasks into a single headline number. The most cited:

# Example: computing Pass@k for HumanEval
import json
import subprocess
import tempfile
import os

def pass_at_k(samples: list[str], test_cases: list[dict], k: int = 1) -> float:
    """
    samples: list of k code completions per problem
    test_cases: list of dicts with 'test' (pytest string) and 'entry_point' (function name)
    """
    pass_counts = 0
    for problem_samples, tc in zip(samples, test_cases):
        passed = 0
        for sample in problem_samples[:k]:
            with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
                f.write(sample + "\n" + tc['test'])
                fname = f.name
            try:
                result = subprocess.run(['python', '-m', 'pytest', fname, '-v'],
                                        capture_output=True, timeout=10)
                if result.returncode == 0:
                    passed += 1
            finally:
                os.unlink(fname)
        if passed > 0:
            pass_counts += 1
    return pass_counts / len(test_cases)

No single benchmark captures production readiness. MMLU correlates with general capability but not with your specific domain. HumanEval measures algorithmic coding, not framework familiarity or debugging. Run benchmarks to track relative progress across model versions, but validate with your own eval set.

8. Calibration and confidence metrics

A well-calibrated model’s predicted probabilities match empirical frequencies: when it says 90% confidence, it’s correct 90% of the time. Calibration matters for: routing (send low-confidence predictions to humans), selective prediction (abstain when uncertain), and risk-sensitive applications.

import numpy as np
from sklearn.calibration import calibration_curve

def expected_calibration_error(probs: np.ndarray, labels: np.ndarray, n_bins: int = 10) -> float:
    """
    probs: predicted probabilities for the positive class (or max class prob for multiclass)
    labels: binary correctness (1 = correct, 0 = incorrect)
    """
    prob_true, prob_pred = calibration_curve(labels, probs, n_bins=n_bins, strategy='uniform')
    bin_counts = np.histogram(probs, bins=n_bins, range=(0, 1))[0]
    ece = np.sum(bin_counts * np.abs(prob_true - prob_pred)) / len(probs)
    return ece

def brier_score(probs: np.ndarray, labels: np.ndarray) -> float:
    return np.mean((probs - labels) ** 2)

ECE (Expected Calibration Error) and Brier score are the standard summaries. Most LLMs are poorly calibrated out of the box — they’re overconfident. Temperature scaling on a validation set helps. If you’re building a system that routes or abstains based on confidence, measure calibration on your actual task distribution, not a generic benchmark.

9. Operational metrics: latency, throughput, and cost

Production evaluation isn’t just quality — it’s the quality-latency-cost Pareto frontier. Key operational metrics:

• TTFT (Time To First Token): latency from request send to first token received. Critical for streaming UX.
• TPOT (Time Per Output Token): inverse of throughput during generation. Determines total latency for long responses.
• Throughput (tokens/sec): aggregate across concurrent requests. GPU utilization proxy.
• Cost per 1k tokens: blended across input/output, provider, and model tier.

# Simple latency measurement wrapper
import time
from dataclasses import dataclass
from typing import Iterator

@dataclass
class LatencyMetrics:
    ttft: float  # seconds
    tpot: float  # seconds per token
    total_tokens: int
    total_time: float

def measure_streaming_latency(stream: Iterator[str]) -> LatencyMetrics:
    start = time.perf_counter()
    first_token_time = None
    tokens = 0
    for chunk in stream:
        if first_token_time is None:
            first_token_time = time.perf_counter()
        tokens += 1
    end = time.perf_counter()
    ttft = first_token_time - start if first_token_time else 0
    total_time = end - start
    tpot = (total_time - ttft) / max(tokens - 1, 1) if tokens > 1 else 0
    return LatencyMetrics(ttft, tpot, tokens, total_time)

Track these per model, per provider, and per request priority tier. A model that scores 2% higher on MMLU but costs 10x more and has 3x TTFT may be the wrong choice for your chat endpoint. n4n.ai surfaces per-token usage metering and provider-level latency histograms so you can make these tradeoffs with data instead of guesses.

10. Robustness and out-of-distribution detection

Evaluation on clean test sets misses failure modes that appear in production: adversarial prompts, distribution shift, multilingual degradation, and context-length extrapolation. Robustness metrics include:

• Adversarial accuracy: accuracy on perturbed inputs (typos, synonym swaps, prompt injections)
• OOD detection AUROC: using model confidence or entropy to detect out-of-distribution inputs
• Length generalization: performance vs. context length (needle-in-haystack, long-context QA)
• Multilingual parity: gap**: performance delta between English and low-resource languages

# Needle-in-haystack test: can the model retrieve a fact buried in context?
def needle_in_haystack(model, tokenizer, context_length: int, needle: str, question: str) -> bool:
    # Build context: filler text + needle at random position + filler
    filler = "The quick brown fox jumps over the lazy dog. " * 100
    tokens = tokenizer.encode(filler)
    insert_pos = len(tokens) // 2
    needle_tokens = tokenizer.encode(needle)
    context_tokens = tokens[:insert_pos] + needle_tokens + tokens[insert_pos:]
    context_tokens = context_tokens[:context_length]
    context = tokenizer.decode(context_tokens)
    prompt = f"{context}\n\nQuestion: {question}\nAnswer:"
    # Generate and check if needle content appears in answer
    answer = model.generate(prompt, max_tokens=50)
    return needle.lower() in answer.lower()

Run these as regression tests in CI. They catch silent degradations when you swap models, update prompts, or change retrieval pipelines.

Summary: choosing your metric stack

Start with one primary metric that reflects your actual user-facing quality bar. Add one operational metric (latency or cost). Run LLM-as-judge on a representative sample weekly. Benchmark suites are for model selection, not production monitoring. The metric that correlates with your business outcome is the only one that ultimately matters — everything else is a proxy.