What is nucleus sampling (top-p) in language models?

Nucleus sampling (top-p) is a decoding strategy that selects the next token from the smallest set of candidates whose cumulative probability exceeds a threshold p. Unlike top-k sampling which fixes the candidate pool size, nucleus sampling adapts the pool size to the model’s confidence at each step.

How nucleus sampling works

At each generation step, the model outputs a probability distribution over the vocabulary. Nucleus sampling sorts tokens by descending probability, then accumulates probability mass until the running total reaches or exceeds p. Only tokens in this accumulated set — the “nucleus” — are eligible for sampling. The selected token is drawn proportionally to its original probability, renormalized within the nucleus.

def nucleus_sampling(logits, top_p=0.9, temperature=1.0):
    """
    logits: raw model output [vocab_size]
    top_p: cumulative probability threshold (0 < top_p <= 1)
    temperature: sharpens (T < 1) or flattens (T > 1) the distribution
    """
    import torch
    
    # Apply temperature
    logits = logits / temperature
    
    # Convert to probabilities
    probs = torch.softmax(logits, dim=-1)
    
    # Sort descending
    sorted_probs, sorted_indices = torch.sort(probs, descending=True)
    
    # Cumulative sum
    cumsum_probs = torch.cumsum(sorted_probs, dim=-1)
    
    # Mask tokens outside the nucleus
    # Keep tokens where cumulative prob <= top_p
    # Also keep the first token that pushes us over the threshold
    nucleus_mask = cumsum_probs <= top_p
    # Ensure at least one token is kept
    nucleus_mask[0] = True
    
    # Zero out excluded tokens
    sorted_probs = sorted_probs * nucleus_mask
    
    # Renormalize
    sorted_probs = sorted_probs / sorted_probs.sum()
    
    # Sample from nucleus
    sampled_idx = torch.multinomial(sorted_probs, num_samples=1)
    token_id = sorted_indices[sampled_idx]
    
    return token_id

The threshold p typically ranges from 0.9 to 0.95. Lower values (0.7–0.8) produce more focused, deterministic output. Higher values (0.95–0.99) approach full distribution sampling, increasing diversity but risking incoherence.

Why top-p matters for generation quality

Fixed top-k sampling suffers from a fundamental mismatch: the model’s uncertainty varies dramatically across positions. After “The capital of France is”, the distribution is sharply peaked on “Paris” — top-k=50 wastes computation on irrelevant tokens. After “The poet wrote a beautiful”, the distribution is flat — top-k=50 may exclude plausible continuations.

Nucleus sampling adapts to this variance. When the model is confident, the nucleus contains few tokens. When uncertain, it expands automatically. This property makes top-p the default choice for most production LLM deployments.

The interaction with temperature is critical. Temperature reshapes the distribution before nucleus selection:

• Temperature < 1.0: Sharpens peaks, shrinking the nucleus for a given p
• Temperature > 1.0: Flattens distribution, expanding the nucleus
• Temperature = 1.0: Uses raw model probabilities

In practice, most systems fix temperature=1.0 and tune top-p, or fix top-p=0.9–0.95 and tune temperature. Tuning both simultaneously creates a two-dimensional search space that’s harder to reason about.

Concrete example

Consider a model completing: “The detective examined the clue and realized”

With temperature=1.0, the top probabilities might be:

With top-p=0.9, the nucleus includes tokens up to “had” (cumulative 0.91). The sampling pool is 8 tokens. “been” and “missed” are excluded despite non-trivial probability.

With top-p=0.95, the nucleus expands to include “been” (cumulative 0.93) and “missed” (0.94), giving 10 tokens.

With top-k=5, only “the”, “it”, “was”, “this”, “meant” are eligible — “something” and “important” are excluded despite combined 0.09 probability.

This illustrates why nucleus sampling preserves more semantic diversity than top-k at equivalent compute budgets.

Common misconceptions

Misconception: top-p and top-k are mutually exclusive

They compose cleanly. Most implementations apply both: first filter by top-k, then by top-p on the remaining tokens. This provides a hard ceiling on candidate count (useful for batching) while retaining adaptive behavior.

def combined_sampling(logits, top_k=50, top_p=0.9, temperature=1.0):
    logits = logits / temperature
    probs = torch.softmax(logits, dim=-1)
    
    # Top-k filter
    top_k_probs, top_k_indices = torch.topk(probs, min(top_k, probs.size(-1)))
    
    # Top-p filter on top-k subset
    sorted_probs, sorted_indices = torch.sort(top_k_probs, descending=True)
    cumsum = torch.cumsum(sorted_probs, dim=-1)
    nucleus_mask = cumsum <= top_p
    nucleus_mask[0] = True
    
    filtered_probs = sorted_probs * nucleus_mask
    filtered_probs = filtered_probs / filtered_probs.sum()
    
    sampled = torch.multinomial(filtered_probs, 1)
    return top_k_indices[sorted_indices[sampled]]

Misconception: top-p=1.0 equals greedy decoding

top-p=1.0 includes the entire vocabulary (minus any top-k cutoff). Sampling from the full distribution is not greedy — it still samples stochastically. Greedy decoding is argmax, equivalent to temperature→0 with any top-p.

Misconception: nucleus sampling eliminates repetition

Nucleus sampling reduces but does not eliminate repetition loops. Repetition arises from the model assigning high probability to recently generated tokens — a modeling issue, not a decoding issue. Repetition penalties (presence/frequency penalties) or contrastive search address this at the logit level, before sampling.

Misconception: higher top-p always means more creative

Beyond ~0.95, increasing top-p adds mostly low-probability noise tokens. The marginal diversity gain diminishes while hallucination risk increases. The sweet spot for most tasks is 0.9–0.95.

When to use nucleus sampling vs alternatives

For structured output (JSON, function calls), avoid sampling entirely. Use constrained decoding or greedy with a grammar constraint. Sampling introduces non-determinism that breaks schema compliance.

Integration notes

When calling an OpenAI-compatible endpoint, the parameters map directly:

{
  "model": "gpt-4o-mini",
  "messages": [...],
  "top_p": 0.9,
  "temperature": 0.7,
  "max_tokens": 500
}

Most providers also support top_k as an extension parameter. If both are provided, the typical server-side logic applies top-k first, then top-p on the truncated distribution.

If you’re routing requests across multiple providers through a gateway, verify that each provider implements nucleus sampling identically. Some older APIs only support top-k, or apply the filters in reverse order. This can cause subtle output distribution shifts when failing over between providers.