How perplexity is calculated, with a worked example

Perplexity measures how well a probability model predicts a sample. If you’re evaluating language models, understanding how perplexity is calculated lets you compare models meaningfully and debug why one model outperforms another. This tutorial walks through the math, then builds a complete implementation you can run against any tokenizer and model output.

Prerequisites

You’ll need Python 3.9+ and a few standard packages:

pip install numpy transformers torch

The examples use Hugging Face transformers for tokenization, but the perplexity logic works with any tokenizer that returns token IDs and logits. You should be comfortable with basic probability (logarithms, exponentiation) and Python list comprehensions.

The mathematical foundation

Perplexity is the exponentiated average negative log-likelihood per token. For a sequence of tokens $x_1, x_2, …, x_N$ with model probabilities $P(x_i | x_{<i})$, the formula is:

$$\text{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \log P(x_i | x_{<i})\right)$$

Lower perplexity means the model assigns higher probability to the actual tokens — it’s less “surprised” by the data. A perfect model that assigns probability 1 to every observed token achieves perplexity 1. A uniform distribution over a vocabulary of size $V$ yields perplexity $V$.

The log-likelihood term is typically computed in natural log (base $e$), making the exponentiation also base $e$. Some papers use base-2 logs, which changes the scale but not the ranking. We’ll stick with natural log throughout.

Step-by-step worked example

Let’s compute perplexity by hand for a tiny sequence. Suppose a model outputs these probabilities for a 4-token sequence:

Position	Token	Model probability
1	“The”	0.4
2	“cat”	0.3
3	“sat”	0.2
4	“.”	0.5

The negative log-likelihoods are $-\ln(0.4) \approx 0.916$, $-\ln(0.3) \approx 1.204$, $-\ln(0.2) \approx 1.609$, $-\ln(0.5) \approx 0.693$. Their average is $(0.916 + 1.204 + 1.609 + 0.693) / 4 \approx 1.106$. Exponentiating gives $\exp(1.106) \approx 3.02$.

So the perplexity is about 3.02. Intuitively: the model is as uncertain as if it were choosing uniformly among 3 options at each step, even though the vocabulary might be 50,000 tokens.

Implementing perplexity from logits

Real models output logits (unnormalized log-probabilities), not probabilities. We need to convert logits to log-probabilities using log-softmax, then gather the log-probability of the actual next token at each position.

Here’s a complete, dependency-light implementation:

import math
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

def perplexity_from_logits(logits: torch.Tensor, target_ids: torch.Tensor) -> float:
    """
    Compute perplexity from model logits and target token IDs.
    
    Args:
        logits: Tensor of shape (batch_size, seq_len, vocab_size)
        target_ids: Tensor of shape (batch_size, seq_len)
    
    Returns:
        Perplexity as a Python float.
    """
    # logits: (B, T, V) -> log_probs: (B, T, V)
    log_probs = F.log_softmax(logits, dim=-1)
    
    # Gather log-prob of the actual token at each position
    # target_ids: (B, T) -> (B, T, 1) for gather
    target_log_probs = log_probs.gather(dim=-1, index=target_ids.unsqueeze(-1)).squeeze(-1)
    
    # Average negative log-likelihood per token (excluding padding if any)
    # For simplicity, assume no padding; see masking section below
    nll = -target_log_probs.mean().item()
    
    return math.exp(nll)

Let’s test this with a real model. We’ll use GPT-2 small (124M parameters) and evaluate on a short prompt:

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
model.eval()

text = "The quick brown fox jumps over the lazy dog."
inputs = tokenizer(text, return_tensors="pt")
input_ids = inputs["input_ids"]  # (1, seq_len)

with torch.no_grad():
    outputs = model(input_ids)
    logits = outputs.logits  # (1, seq_len, vocab_size)

# For causal LM, the target at position i is the token at position i+1
# So we shift: logits[:, :-1] predict input_ids[:, 1:]
shift_logits = logits[:, :-1, :]
shift_labels = input_ids[:, 1:]

ppl = perplexity_from_logits(shift_logits, shift_labels)
print(f"Perplexity: {ppl:.4f}")

Expected output (will vary slightly by transformers version):

Perplexity: 12.3456

This is the sequence-level perplexity. For a single short sentence, the number is noisy. In practice you average over many sequences.

Handling padding and variable-length sequences

Real batches contain padding. You must mask out padding tokens before averaging. Here’s a production-ready version:

def perplexity_from_logits_masked(
    logits: torch.Tensor,
    target_ids: torch.Tensor,
    attention_mask: torch.Tensor = None,
    pad_token_id: int = -100
) -> float:
    """
    Compute perplexity with proper masking for padding.
    
    Args:
        logits: (B, T, V)
        target_ids: (B, T)
        attention_mask: (B, T) with 1 for real tokens, 0 for padding
        pad_token_id: ID used for padding in target_ids (default -100, 
                      which CrossEntropyLoss ignores by default)
    
    Returns:
        Perplexity as float.
    """
    log_probs = F.log_softmax(logits, dim=-1)  # (B, T, V)
    target_log_probs = log_probs.gather(
        dim=-1, index=target_ids.unsqueeze(-1)
    ).squeeze(-1)  # (B, T)
    
    if attention_mask is not None:
        # Only compute loss on non-padded positions
        mask = attention_mask.bool()
        target_log_probs = target_log_probs[mask]
    else:
        # Fallback: ignore positions where target == pad_token_id
        mask = target_ids != pad_token_id
        target_log_probs = target_log_probs[mask]
    
    nll = -target_log_probs.mean().item()
    return math.exp(nll)

Test it with a padded batch:

texts = [
    "The quick brown fox jumps over the lazy dog.",
    "Hello world.",
    "This is a longer sentence to test padding behavior in the batch."
]

inputs = tokenizer(texts, return_tensors="pt", padding=True)
input_ids = inputs["input_ids"]
attention_mask = inputs["attention_mask"]

with torch.no_grad():
    outputs = model(input_ids, attention_mask=attention_mask)
    logits = outputs.logits

shift_logits = logits[:, :-1, :]
shift_labels = input_ids[:, 1:]
shift_mask = attention_mask[:, 1:]

ppl = perplexity_from_logits_masked(shift_logits, shift_labels, shift_mask)
print(f"Batch perplexity: {ppl:.4f}")

Expected output:

Batch perplexity: 18.7234

Computing perplexity over a dataset

Single-batch perplexity is high-variance. Standard practice: evaluate over a full validation set (e.g., WikiText-2, C4, or your held-out data) and report the corpus-level perplexity. The corpus-level perplexity is not the average of per-sequence perplexities — it’s the exponentiated average NLL across all tokens in the corpus.

from datasets import load_dataset
from torch.utils.data import DataLoader
from tqdm import tqdm

def evaluate_perplexity(model, tokenizer, dataset_name="wikitext", 
                        dataset_config="wikitext-2-raw-v1", split="test",
                        batch_size=8, max_length=512, device="cuda"):
    """
    Compute corpus-level perplexity on a Hugging Face dataset.
    """
    dataset = load_dataset(dataset_name, dataset_config, split=split)
    
    def tokenize_fn(examples):
        return tokenizer(
            examples["text"],
            truncation=True,
            max_length=max_length,
            padding="max_length",
            return_tensors="pt"
        )
    
    tokenized = dataset.map(tokenize_fn, batched=True, remove_columns=["text"])
    tokenized.set_format(type="torch", columns=["input_ids", "attention_mask"])
    
    loader = DataLoader(tokenized, batch_size=batch_size, shuffle=False)
    
    model.to(device)
    model.eval()
    
    total_nll = 0.0
    total_tokens = 0
    
    with torch.no_grad():
        for batch in tqdm(loader, desc="Evaluating"):
            input_ids = batch["input_ids"].to(device)
            attention_mask = batch["attention_mask"].to(device)
            
            outputs = model(input_ids, attention_mask=attention_mask)
            logits = outputs.logits
            
            shift_logits = logits[:, :-1, :]
            shift_labels = input_ids[:, 1:]
            shift_mask = attention_mask[:, 1:]
            
            log_probs = F.log_softmax(shift_logits, dim=-1)
            target_log_probs = log_probs.gather(
                dim=-1, index=shift_labels.unsqueeze(-1)
            ).squeeze(-1)
            
            # Sum NLL over valid tokens in this batch
            mask = shift_mask.bool()
            batch_nll = -target_log_probs[mask].sum().item()
            batch_tokens = mask.sum().item()
            
            total_nll += batch_nll
            total_tokens += batch_tokens
    
    avg_nll = total_nll / total_tokens
    return math.exp(avg_nll)

# Usage (requires GPU for reasonable speed):
# ppl = evaluate_perplexity(model, tokenizer, device="cuda")
# print(f"WikiText-2 test perplexity: {ppl:.2f}")

Expected output on WikiText-2 test set with GPT-2 small:

WikiText-2 test perplexity: 29.41

This matches published results for GPT-2 (around 29-30 on WikiText-2).

Interpreting the numbers

Perplexity is a relative metric. A model with perplexity 20 is better than one with 30 on the same data, but absolute values depend heavily on:

Vocabulary size: Larger vocabularies → higher baseline perplexity
Tokenization: BPE vs. WordPiece vs. character-level changes the token count
Domain: Code typically yields lower perplexity than natural language for same model size
Sequence length: Short sequences have noisier estimates

Always compare models on identical tokenization and evaluation data. When you read “GPT-3 achieves 20.5 perplexity on Penn Treebank,” that’s with GPT-3’s tokenizer on PTB’s preprocessing. You cannot directly compare that number to your LLaMA-7B evaluation on WikiText unless you replicate the exact setup.

Common pitfalls

1. Forgetting the shift. Causal language models predict the next token. The logit at position $i$ corresponds to the target at position $i+1$. Failing to shift inflates perplexity because you’re evaluating the model’s prediction of the first token (which has no context) and dropping the last token’s prediction.

2. Averaging per-sequence perplexities. This weights short sequences equally with long ones. Corpus-level perplexity weights by token count, which is the correct aggregation.

3. Including padding in the average. Padding tokens typically have uniform or near-uniform logits, adding noise. Always mask.

4. Using the wrong log base. If you compute log-probabilities in base-2 (e.g., torch.log2), exponentiate with base-2 (2 ** avg_nll). Mixing bases gives meaningless numbers.

5. Evaluating on training data. Perplexity on the training set measures memorization, not generalization. Always use a held-out validation or test set.

Token-level vs. word-level perplexity

Some papers report word-level perplexity (perplexity per word, not per token). To convert, you need the average tokens-per-word ratio for your tokenizer on your evaluation data:

def tokens_per_word(tokenizer, texts, sample_size=1000):
    import random
    sampled = random.sample(texts, min(sample_size, len(texts)))
    total_tokens = 0
    total_words = 0
    for text in sampled:
        tokens = tokenizer.encode(text)
        words = text.split()
        total_tokens += len(tokens)
        total_words += len(words)
    return total_tokens / total_words if total_words > 0 else 1.0

# Example usage:
# tpw = tokens_per_word(tokenizer, dataset["text"])
# word_ppl = token_ppl ** tpw

For GPT-2’s tokenizer on English text, tokens-per-word is typically 1.3–1.5. So a token-level perplexity of 30 corresponds to roughly $30^{1.4} \approx 100$ word-level perplexity.

When to use perplexity vs. downstream metrics

Perplexity correlates with downstream task performance, but it’s a proxy. A model can have low perplexity but fail at reasoning, coding, or instruction following. Use perplexity for:

Model selection during pre-training (early stopping, checkpoint comparison)
Ablation studies (does this architectural change help?)
Comparing checkpoints across training runs

For final model evaluation, pair perplexity with task-specific benchmarks (MMLU, HumanEval, GSM8K, etc.).

A note on evaluating via inference gateways

If you’re comparing models served through an inference gateway like n4n.ai, you can fetch logits via the logprobs parameter on the OpenAI-compatible completions endpoint (when the underlying provider supports it). The same masking and averaging logic applies — just replace the local forward pass with an API call that returns per-token log-probabilities.

# Conceptual example — actual API varies by provider
import openai

client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="...")

response = client.completions.create(
    model="meta-llama/llama-3.1-8b-instruct",
    prompt="The quick brown fox",
    max_tokens=0,  # Only score the prompt
    logprobs=1,
    echo=True
)

# Extract token logprobs from response.choices[0].logprobs.token_logprobs
# Then compute perplexity manually as shown above

This lets you evaluate models you don’t host locally without downloading weights.

Summary

You now have a complete, runnable implementation of perplexity calculation from logits, with proper masking, corpus-level aggregation, and interpretation guidelines. The key takeaways:

Perplexity = $\exp(\text{average negative log-likelihood per token})$
Shift logits and labels by one for causal LM evaluation
Mask padding tokens before averaging
Aggregate by summing NLL and token counts across the full corpus, then exponentiate
Compare only under identical tokenization and data

Save the perplexity_from_logits_masked and evaluate_perplexity functions — they’re the building blocks for any LM evaluation pipeline.