How top-p and temperature work together

Understanding how top-p and temperature together shape model outputs is essential for anyone shipping LLM-powered features. These two parameters control fundamentally different aspects of the sampling process: temperature adjusts the probability distribution before selection, while top-p truncates the vocabulary after that adjustment. Most engineers treat them as independent knobs, but their interaction creates non-obvious behaviors that can make or break production reliability.

The mechanics: what each parameter actually does

Temperature rescales the logits before the softmax. A temperature of 1.0 leaves the distribution unchanged. Values below 1.0 sharpen it — the model becomes more confident, concentrating probability mass on the top tokens. Values above 1.0 flatten it, spreading probability toward the tail. Mathematically:

def apply_temperature(logits, temperature):
    if temperature == 0:
        return logits  # argmax handled separately
    return logits / temperature

Top-p (nucleus sampling) operates on the resulting probability distribution. It sorts tokens by probability, accumulates them until the cumulative mass exceeds p, then renormalizes and samples only from that nucleus. Everything outside gets zero probability.

def top_p_filter(probs, p):
    sorted_indices = torch.argsort(probs, descending=True)
    sorted_probs = probs[sorted_indices]
    cumulative = torch.cumsum(sorted_probs, dim=-1)
    mask = cumulative <= p
    # Always keep at least the top token
    mask[0] = True
    filtered = torch.zeros_like(probs)
    filtered[sorted_indices[mask]] = sorted_probs[mask]
    return filtered / filtered.sum()

The critical insight: temperature runs first, then top-p filters the result. Changing temperature changes which tokens fall inside the top-p nucleus.

Why the order matters

Because temperature reshapes the distribution before top-p applies, the same top-p value behaves differently at different temperatures. At low temperature, the distribution is already peaked — top-p=0.9 might only admit 2-3 tokens. At high temperature, the distribution is flat, so top-p=0.9 could admit 50+ tokens.

This creates a practical trap: engineers often set temperature=0.7 and top-p=0.9 as “reasonable defaults” without realizing that top-p is doing almost nothing at that temperature. The nucleus is tiny because the distribution is already concentrated.

# At temperature 0.7, top-p=0.9 typically keeps ~3-5 tokens
# At temperature 1.2, top-p=0.9 typically keeps ~30-60 tokens
# At temperature 0.2, top-p=0.9 typically keeps 1-2 tokens (effectively greedy)

Practical parameter combinations

Deterministic, factual output

{
  "temperature": 0,
  "top_p": 1.0
}

Temperature 0 means argmax — the single highest-probability token always wins. Top-p is irrelevant but harmless. Use for classification, extraction, code generation with strict specs, any task where consistency beats creativity.

Controlled creativity with guardrails

{
  "temperature": 0.7,
  "top_p": 0.9
}

The classic “chat” default. Temperature 0.7 softens the distribution enough to allow variation. Top-p=0.9 clips the extreme tail (hallucination-prone low-probability tokens) while keeping meaningful alternatives. Works well for general conversation, summarization, brainstorming.

High diversity, coherent narratives

{
  "temperature": 1.0,
  "top_p": 0.95
}

Temperature 1.0 uses the model’s native distribution. Top-p=0.95 removes only the most improbable tokens. Good for creative writing, story generation, roleplay where you want the model’s full expressive range but still filter nonsense.

Maximum diversity (use carefully)

{
  "temperature": 1.2,
  "top_p": 0.99
}

High temperature flattens aggressively. Near-1.0 top-p keeps almost everything. Expect incoherence, topic drift, and occasional garbage tokens. Only appropriate for pure exploration or when you’re post-filtering with a classifier.

Common pitfalls

Pitfall 1: Setting both high expecting “more randomness”

{ "temperature": 1.5, "top_p": 0.9 }

This doesn’t give you “maximum creativity.” High temperature spreads probability mass thin. Top-p=0.9 then admits a huge nucleus — sometimes hundreds of tokens. The model samples from a vast pool of nearly-equally-likely garbage. Output becomes word salad.

Fix: If you want diversity, raise temperature or raise top-p, not both. Start with temperature=1.0, top-p=0.95 and adjust one at a time.

Pitfall 2: Low temperature with low top-p

{ "temperature": 0.3, "top_p": 0.5 }

Temperature 0.3 already concentrates mass on the top few tokens. Top-p=0.5 then aggressively truncates, often leaving only the single argmax token. You’ve effectively recreated temperature=0 with extra steps — but less predictable because the cutoff depends on the exact probability distribution.

Fix: At temperature < 0.5, set top-p=1.0 (disabled). Let temperature do the work alone.

Pitfall 3: Ignoring top-k

Many APIs also support top-k (keep only the k highest-probability tokens). The typical pipeline order: temperature → top-k → top-p. If you set top-k=50 and top-p=0.9, top-k runs first and may eliminate tokens that would have been in the top-p nucleus.

# Typical sampling pipeline order
logits = model(input_ids)
logits = logits / temperature
logits = top_k_filter(logits, k)      # if k > 0
probs = softmax(logits)
probs = top_p_filter(probs, p)        # if p < 1.0
token = sample(probs)

Fix: If using both, set top-k high enough (100-200) that it rarely binds, or disable one. Most production systems pick one truncation method.

Pitfall 4: Assuming defaults are safe

OpenAI’s defaults: temperature=1.0, top-p=1.0 (both disabled). Anthropic’s defaults vary by model. Open-source servers (vLLM, TGI, Ollama) often default to temperature=0.7, top-p=0.9. If you’re switching providers or models, explicitly set both parameters. Implicit defaults are a deployment risk.

Tuning workflow for production

1. Start with temperature only. Set top-p=1.0. Sweep temperature: 0.0, 0.3, 0.5, 0.7, 1.0. Evaluate outputs for your task. Find the highest temperature that maintains acceptable quality.

2. Add top-p only if needed. If you see occasional hallucinated tokens or nonsensical continuations at your chosen temperature, lower top-p to 0.95, then 0.9. Stop when the artifacts disappear.

3. Lock both. Document the pair. Treat them as a single configuration unit — changing one requires re-evaluation.

4. Add monitoring. Log the effective vocabulary size (number of tokens with non-zero probability after both filters) per request. Sudden drops indicate distribution collapse; sudden spikes indicate temperature drift or prompt injection.

def effective_vocab_size(logits, temperature, top_p, top_k=0):
    logits = logits / temperature if temperature > 0 else logits
    if top_k > 0:
        logits = top_k_filter(logits, top_k)
    probs = softmax(logits)
    if top_p < 1.0:
        probs = top_p_filter(probs, top_p)
    return (probs > 0).sum().item()

When to use which parameter

A note on provider behavior

Some inference gateways normalize or clamp these parameters. For example, n4n.ai forwards temperature and top-p directly to upstream providers but logs the effective sampling configuration per request so you can debug distribution shifts across model versions. If you’re routing across multiple providers, verify each one’s parameter bounds — some clamp temperature to [0, 2], others to [0, 1.5]; some ignore top-p > 0.99.

Summary checklist

• Temperature runs first, reshaping the distribution; top-p runs second, truncating the result
• Low temperature + high top-p = top-p does nothing
• High temperature + low top-p = huge nucleus, word salad risk
• Don’t tune both simultaneously; fix one, sweep the other
• Disable top-p (set to 1.0) when temperature < 0.5
• Explicitly set both in every API call — never rely on defaults
• Monitor effective vocabulary size in production
• Document the temperature/top-p pair as a single config unit

The interaction between top-p and temperature together is predictable once you internalize the pipeline order. Most production issues stem from treating them as independent creativity dials rather than sequential filters. Set a workflow, log the effective vocab, and you’ll avoid the common failure modes.