Combining top-p, top-k and temperature: a practical guide

Combining top-p top-k temperature gives you fine-grained control over the randomness and diversity of LLM outputs. Most engineers reach for temperature alone, but the three parameters interact in a specific order that changes how each one behaves. Understanding that interaction lets you dial in exactly the behavior you need — whether that’s deterministic code generation, creative writing, or something in between.

How each parameter works alone

Before combining them, it helps to see what each one does in isolation. All three operate on the logits (unnormalized log-probabilities) the model produces for the next token.

Temperature scales the logits before softmax. A temperature of 1.0 leaves logits unchanged. Values below 1.0 sharpen the distribution (more confident, less random). Values above 1.0 flatten it (more random, more diverse).

def apply_temperature(logits: torch.Tensor, temperature: float) -> torch.Tensor:
    if temperature == 0:
        # Greedy decoding — argmax
        return torch.zeros_like(logits).scatter_(-1, logits.argmax(-1, keepdim=True), 1.0)
    return logits / temperature

Top-k keeps only the k most likely tokens and masks the rest to negative infinity before softmax. This prevents the model from ever selecting extremely unlikely tokens, regardless of temperature.

def apply_top_k(logits: torch.Tensor, k: int) -> torch.Tensor:
    if k <= 0:
        return logits
    values, indices = torch.topk(logits, k, dim=-1)
    min_value = values[:, -1:]  # threshold = k-th highest logit
    return torch.where(logits < min_value, float('-inf'), logits)

Top-p (nucleus sampling) keeps the smallest set of tokens whose cumulative probability exceeds p. This adapts to the shape of the distribution — tight distributions keep fewer tokens, flat distributions keep more.

def apply_top_p(logits: torch.Tensor, p: float) -> torch.Tensor:
    if p >= 1.0:
        return logits
    probs = torch.softmax(logits, dim=-1)
    sorted_probs, sorted_indices = torch.sort(probs, descending=True, dim=-1)
    cumulative_probs = torch.cumsum(sorted_probs, dim=-1)
    # Mask tokens where cumulative prob exceeds p
    sorted_mask = cumulative_probs > p
    # Keep at least one token
    sorted_mask[:, 0] = False
    # Scatter mask back to original indices
    mask = torch.zeros_like(logits, dtype=torch.bool).scatter_(-1, sorted_indices, sorted_mask)
    return logits.masked_fill(mask, float('-inf'))

The interaction order matters

The standard pipeline applies these in a fixed sequence: temperature → top-k → top-p → sample. This order is not arbitrary — each step reshapes the distribution that the next step sees.

def sample_next_token(
    logits: torch.Tensor,
    temperature: float = 1.0,
    top_k: int = 0,
    top_p: float = 1.0,
) -> torch.Tensor:
    # 1. Temperature scaling
    logits = logits / temperature if temperature > 0 else logits
    
    # 2. Top-k filtering
    if top_k > 0:
        logits = apply_top_k(logits, top_k)
    
    # 3. Top-p filtering
    if top_p < 1.0:
        logits = apply_top_p(logits, top_p)
    
    # 4. Sample from final distribution
    probs = torch.softmax(logits, dim=-1)
    return torch.multinomial(probs, num_samples=1)

Because temperature runs first, it controls the shape of the distribution before truncation. A high temperature flattens the distribution, which means top-p retains more tokens (since cumulative probability spreads across more candidates). A low temperature sharpens the distribution, so top-p retains fewer.

Top-k runs second, imposing a hard ceiling on vocabulary size regardless of probability mass. This can cut off tokens that top-p would have kept, or keep tokens that top-p would have dropped — whichever is more restrictive wins.

Top-p runs last, making the final adaptive cut on the already-filtered distribution.

Practical combinations for common use cases

Deterministic code generation and extraction

{
  "temperature": 0,
  "top_k": 1,
  "top_p": 1.0
}

Temperature 0 makes the model greedy (argmax). Top-k=1 reinforces this. Top-p is irrelevant but harmless. Use this for SQL generation, JSON extraction, classification, and any task where you want the single most likely answer.

Low-diversity creative writing

{
  "temperature": 0.3,
  "top_k": 50,
  "top_p": 0.9
}

Low temperature keeps the model focused. Top-k=50 prevents the long tail of nonsense tokens. Top-p=0.9 allows some flexibility while cutting the extreme tail. Good for summarization, translation, and structured content where you want minor variation but not surprises.

Balanced general-purpose chat

{
  "temperature": 0.7,
  "top_k": 40,
  "top_p": 0.95
}

This is a common default for chat assistants. Temperature 0.7 provides noticeable variety without incoherence. Top-k=40 caps vocabulary per step. Top-p=0.95 is permissive — it mostly lets temperature and top-k do the work, only cutting truly negligible tokens.

High-diversity brainstorming

{
  "temperature": 1.0,
  "top_k": 100,
  "top_p": 0.9
}

Full temperature preserves the model’s native uncertainty. Top-k=100 allows a wide vocabulary. Top-p=0.9 still trims the extreme tail. Use for idea generation, story branching, or when you want the model to “think outside the box.”

Maximum diversity with guardrails

{
  "temperature": 1.2,
  "top_k": 0,
  "top_p": 0.92
}

Temperature above 1.0 amplifies low-probability tokens. Disabling top-k (0 or -1 depending on API) lets top-p do all the truncation adaptively. Top-p=0.92 prevents the model from sampling completely incoherent tokens while still allowing wild exploration. Use sparingly — this often produces hallucinations.

Common pitfalls

Setting temperature too high with top-p

A frequent mistake: temperature=1.5, top_p=0.9. The high temperature flattens the distribution so much that top-p retains a huge number of tokens — often hundreds. The result is near-random sampling with a thin veneer of probability filtering. If you want high diversity, raise temperature and lower top-p together, or rely on top-k to enforce a hard cap.

Using top-k without top-p (or vice versa) unintentionally

Many APIs default top-k to 0 (disabled) and top-p to 1.0 (disabled). If you set only one, you may get behavior you didn’t expect. For example, temperature=0.7, top_k=0, top_p=0.9 means top-p does all the truncation. That’s fine — but be explicit about what you’re disabling.

// Explicit: only top-p active
{
  "temperature": 0.7,
  "top_k": 0,
  "top_p": 0.9
}

// Explicit: only top-k active
{
  "temperature": 0.7,
  "top_k": 40,
  "top_p": 1.0
}

Assuming temperature 0 equals top-k 1

At temperature 0, the model is greedy — it picks the single highest-logit token. Adding top_k=1 changes nothing. Adding top_p=0.5 also changes nothing (the top token has cumulative probability 1.0). But some APIs treat temperature=0 as a special case that bypasses sampling entirely, while others run the full pipeline. Test your provider.

Ignoring the minimum token requirement

Both top-k and top-p implementations typically guarantee at least one token survives. But if you set top_k=1 and top_p=0.01 on a distribution where the top token has probability 0.009, you get undefined behavior — some implementations keep the top token anyway, others error. Keep top-p ≥ 0.01 and top-k ≥ 1 when active.

Forgetting that logit processors run per-token

These parameters apply at every generation step. A setting that works for the first token may produce different behavior ten tokens in, as the context changes the logit distribution. This is why long generations sometimes “go off the rails” even with conservative settings — the distribution shifts, and your fixed thresholds now admit different tokens.

Debugging and tuning workflow

When the output isn’t what you want, follow this sequence:

1. Start with temperature only

{ "temperature": 0.7, "top_k": 0, "top_p": 1.0 }

Generate 5-10 samples. If they’re too random, lower temperature. If too repetitive, raise it. Find the temperature that gives the right baseline diversity.

2. Add top-p to trim the tail

{ "temperature": 0.7, "top_k": 0, "top_p": 0.9 }

Lower top-p until you stop seeing obvious nonsense tokens (repetition loops, non-sequiturs, garbled text). Typical range: 0.85-0.98. If you go below 0.8, you’re likely cutting meaningful diversity.

3. Add top-k only if needed

{ "temperature": 0.7, "top_k": 40, "top_p": 0.9 }

Top-k is a blunt instrument. Add it only if top-p alone lets through tokens you don’t want, or if you need a hard vocabulary cap for latency/throughput reasons (smaller top-k = faster sampling on some kernels).

4. Verify with logprobs

Request logprobs=true (or return_logprobs depending on API) and inspect the actual sampled token probabilities. You’ll see exactly what the pipeline produced.

# Example: inspect what top-p actually kept
def debug_sampling(logits, temperature, top_k, top_p):
    scaled = logits / temperature
    if top_k > 0:
        scaled = apply_top_k(scaled, top_k)
    if top_p < 1.0:
        scaled = apply_top_p(scaled, top_p)
    probs = torch.softmax(scaled, dim=-1)
    top_probs, top_indices = torch.topk(probs, 20)
    print("Top 20 tokens after filtering:")
    for p, idx in zip(top_probs[0], top_indices[0]):
        print(f"  {tokenizer.decode([idx])}: {p:.4f}")

Look for:

• How many tokens have non-zero probability
• Whether the sampled token is in the top 5 (it usually should be)
• Whether probability mass is concentrated or diffuse

5. Stress-test edge cases

Feed the model prompts that produce:

• Very sharp distributions (factual recall, code completion)
• Very flat distributions (open-ended “continue this story”)
• Repetitive loops (“the the the the”)
• High-entropy contexts (long conversations)

Verify your settings don’t break in any of these regimes.

Provider-specific behavior notes

The OpenAI API exposes temperature and top_p but not top_k. Their documentation recommends using only one of temperature or top-p, not both — but many practitioners combine them anyway. If you set both, temperature runs first, then top-p.

Anthropic’s API exposes all three: temperature, top_p, and top_k. They apply in the standard order.

Open-source servers (vLLM, TGI, Ollama) typically expose all three and follow the standard pipeline. Some allow additional logit processors (repetition penalty, frequency/presence penalty) that run after top-p but before sampling.

When routing requests across multiple providers through a gateway, be aware that default values differ. Explicitly set all three parameters in your request if you need consistent behavior. For example, n4n.ai forwards sampling parameters directly to the underlying provider, so a request with temperature=0.7, top_p=0.9, top_k=40 behaves identically to calling that provider directly — but you should still verify each provider’s defaults if you omit any parameter.

Quick reference card

Summary

Combining top-p top-k temperature works because each parameter constrains a different aspect of the sampling distribution. Temperature shapes the curve. Top-k enforces a hard vocabulary ceiling. Top-p adapts to the probability mass. Applied in that order, they give you a three-dimensional control surface.

Start with temperature alone. Add top-p to trim the tail. Add top-k only when you need a hard cap. Verify with logprobs. Test edge cases. And always set all three explicitly when crossing provider boundaries — defaults are not your friend.