Top-p vs top-k sampling: what's the difference?

When you tune an LLM’s sampling parameters, the choice between top-p vs top-k sampling is one of the first decisions that materially changes output quality. Both methods truncate the vocabulary before sampling, but they draw the line in fundamentally different ways: top-p uses a cumulative probability threshold, while top-k uses a fixed token count. Understanding that difference determines whether your model produces coherent prose, diverse code completions, or hallucinated nonsense.

What top-p sampling does

Top-p (nucleus sampling) sorts the vocabulary by descending probability and keeps the smallest set of tokens whose cumulative probability exceeds the threshold p. If p = 0.9, the model samples from the “nucleus” containing 90% of the probability mass. The number of tokens in that nucleus varies by context — sometimes 5 tokens, sometimes 200.

def top_p_logits(logits: torch.Tensor, p: float) -> torch.Tensor:
    """Mask logits outside the top-p nucleus."""
    probs = torch.softmax(logits, dim=-1)
    sorted_probs, sorted_indices = torch.sort(probs, descending=True)
    cumsum_probs = torch.cumsum(sorted_probs, dim=-1)
    
    # Keep tokens where cumulative prob <= p
    # Always keep at least the top token
    mask = cumsum_probs <= p
    mask[..., 0] = True
    
    # Scatter back to original indices
    filtered_logits = torch.full_like(logits, float('-inf'))
    filtered_logits.scatter_(-1, sorted_indices, 
                              torch.where(mask, sorted_probs, float('-inf')))
    return filtered_logits

The adaptive width is the defining characteristic. On a sharp distribution (the model is confident), top-p keeps few tokens. On a flat distribution (the model is uncertain), it keeps many. This mirrors how humans reason: when you’re sure, you don’t hedge; when you’re uncertain, you consider more alternatives.

What top-k sampling does

Top-k keeps exactly k highest-probability tokens and zeros out the rest. If k = 50, you always sample from the top 50 tokens, whether they represent 99% or 10% of the probability mass.

def top_k_logits(logits: torch.Tensor, k: int) -> torch.Tensor:
    """Mask all but the top-k logits."""
    values, indices = torch.topk(logits, k, dim=-1)
    min_value = values[..., -1:].expand_as(logits)
    return torch.where(logits >= min_value, logits, float('-inf'))

The fixed width makes top-k predictable and cheap to implement. It’s the default in many older codebases (including the original GPT-2 release) because it requires no cumulative sum or sorting of probabilities — just a single topk call.

Comparison across dimensions

How they behave in practice

Sharp distributions: the model knows the answer

When the next token is obvious — closing a parenthesis, completing a keyword, finishing a common phrase — the probability distribution is sharply peaked. Top-p with p = 0.9 might keep only 2–3 tokens. Top-k with k = 50 keeps 50 tokens, 47 of which have near-zero probability.

This is where top-p shines. It effectively becomes greedy decoding without the brittleness of temperature=0. You get deterministic-feeling output where determinism is warranted, but the model can still branch if the context genuinely warrants it.

Top-k here wastes compute sampling from noise. It also introduces unnecessary variance: those 47 near-zero tokens occasionally get picked due to floating-point noise or temperature scaling, producing rare but visible glitches.

Flat distributions: the model is uncertain

Open-ended generation — creative writing, brainstorming, the first token of a function body — produces flatter distributions. Top-p expands the nucleus to maintain 90% coverage, sometimes admitting hundreds of tokens. Top-k stubbornly caps at 50.

Here top-k acts as a stronger constraint. It forces the model to choose from a tighter set, which can reduce meandering but also cuts off legitimate long-tail options. Top-p preserves more diversity, which matters for creative tasks but can increase hallucination risk in factual contexts.

The long-tail problem

Real vocabulary distributions follow Zipf’s law. The top 100 tokens might cover 80% of probability mass; the next 10,000 cover the remaining 20%. Top-p at 0.9 will sometimes reach deep into that tail. Top-k at 50 never does.

This has concrete consequences for code generation. Consider a completion where the correct token is a rare variable name (user_authentication_token vs token). Top-k=50 will never see it. Top-p=0.95 might, if the model assigns it enough probability. But top-p also admits more garbage tokens from the tail.

Interaction with temperature

Temperature and top-p/top-k compose non-linearly. Temperature sharpens (T < 1) or flattens (T > 1) the distribution before truncation. This changes how many tokens survive the filter.

def sample_with_temp_top_p(logits: torch.Tensor, temp: float, p: float) -> int:
    scaled = logits / temp
    filtered = top_p_logits(scaled, p)
    probs = torch.softmax(filtered, dim=-1)
    return torch.multinomial(probs, 1).item()

With low temperature (0.2–0.5), distributions sharpen dramatically. Top-p often collapses to 1–3 tokens, making the p value nearly irrelevant. Top-k still keeps k tokens, but the bottom k-1 have vanishing probability — effectively wasted computation.

With high temperature (1.0–1.5), distributions flatten. Top-p expands aggressively. Top-k becomes the binding constraint. If you run T=1.2 with top-k=40, you’re sampling from a uniform-ish distribution over 40 tokens — essentially random choice from a curated set.

Practical rule: Pair low temperature with top-p. Pair high temperature with top-k if you want a hard diversity cap, or top-p if you want probability-mass fidelity.

Which to choose

Default for general-purpose chat and instruction following: top-p = 0.9–0.95

This is the safe default for most production LLM serving. It adapts to the model’s confidence, preserves coherence on sharp distributions, and allows diversity where the model is genuinely uncertain. The p parameter has intuitive semantics (“keep 90% of the probability mass”) that transfers across models and vocabularies.

At n4n.ai we see this configuration serve 80%+ of traffic well across 240+ models without per-model tuning.

Code completion with strict syntax: top-p = 0.95, temperature = 0.2

Low temperature sharpens the distribution; top-p at 0.95 keeps just enough tokens to handle identifier names and rare-but-correct tokens. Avoid top-k here — it arbitrarily caps the candidate set and can exclude the right variable name.

Creative writing and brainstorming: top-p = 0.9–1.0, temperature = 0.8–1.0

Higher temperature flattens the distribution. Top-p at 1.0 disables truncation entirely (sample from full vocabulary). If you want a safety rail, top-p = 0.95 still admits the long tail while excluding the absolute garbage tier.

Classification and structured extraction: top-p = 0.1–0.3, temperature = 0.0–0.1

You want near-deterministic output. Low top-p acts as a soft constraint that still allows the model to self-correct if the top token is clearly wrong (e.g., a typo in the training data). Pure greedy (temperature=0) is brittle; top-p=0.1 with temperature=0.1 is robust.

Legacy compatibility and reproducible benchmarks: top-k = 40–50

If you’re reproducing a paper that used top-k, or integrating with a system that hardcodes top-k logic, match it. The fixed k makes results reproducible across implementations — no cumulative-sum floating-point differences.

Latency-critical paths with small models: top-k = 20–30

On small models (1–3B parameters) running on CPU or edge devices, the O(V log V) sort for top-p can dominate sampling latency. A fused top-k kernel (O(V log k)) is measurably faster. The quality hit is acceptable for constrained vocabularies.

One more thing: don’t use both simultaneously

Some APIs let you set both top-p and top-k. The typical implementation applies top-k first, then top-p on the survivors. This is rarely what you want — it creates a confusing interaction where the effective p depends on k and the distribution shape. Pick one, tune it, move on.

If you need a hard cap and probability-mass fidelity, implement a custom sampler: top-p with a max-token fallback. But 99% of the time, top-p alone handles both concerns.