Most engineers treat temperature and top-p as independent knobs, but they interact in ways that silently break output quality. When you set both, the effective sampling distribution becomes the intersection of two truncation strategies — and the order of operations matters. This guide walks through the mechanics, shows which combinations actually work in production, and gives you a repeatable tuning process.
How each parameter works alone
Temperature rescales the logits before softmax. A temperature of 1.0 leaves the model’s native distribution untouched. Values below 1.0 sharpen the distribution (more peaky, less entropy); values above 1.0 flatten it (more uniform, more entropy). At 0.0, you get deterministic argmax — the single highest-probability token every time.
Top-p (nucleus sampling) works after temperature scaling. It sorts tokens by probability, accumulates mass until it hits the threshold p, then renormalizes and samples only from that nucleus. A top-p of 0.9 means “sample from the smallest set of tokens that together account for 90% of the probability mass.”
The critical detail: temperature applies first, then top-p truncates the result. This ordering is universal across OpenAI, Anthropic, and open-source implementations.
The interaction mechanics
When you combine temperature and top-p together, you get a two-stage filter:
logits → temperature scaling → softmax → top-p truncation → renormalize → sampleLow temperature + low top-p is redundant. If temperature = 0.2 already concentrates 95% of mass on the top 3 tokens, a top-p of 0.9 changes nothing — the nucleus already contains those tokens. You’re paying the computational cost of sorting and accumulation for no behavioral difference.
High temperature + high top-p is dangerous. Temperature = 1.5 spreads mass across hundreds of tokens. Top-p = 0.95 then admits a huge nucleus, including low-probability tokens that the model never intended to be viable. You get fluent nonsense: grammatically correct but semantically unmoored.
The sweet spot is moderate temperature with restrictive top-p, or low temperature with generous top-p. Both configurations keep the nucleus small and meaningful.
Practical parameter combinations
Deterministic extraction (classification, entity extraction, JSON mode)
temperature: 0.0
top_p: 1.0 (or omit)Temperature 0.0 makes the model deterministic. Top-p is irrelevant because there’s no sampling — argmax always wins. Some APIs still require a top-p value; pass 1.0 to disable nucleus filtering.
Factual QA with slight variation
temperature: 0.2
top_p: 0.9Low temperature keeps the model on the high-probability ridge. Top-p 0.9 admits just enough alternatives to avoid repetitive loops on edge cases. This is the default for most production RAG pipelines.
Creative writing with guardrails
temperature: 0.7
top_p: 0.9Standard “creative” preset. Temperature 0.7 allows surprising but plausible continuations. Top-p 0.9 cuts the long tail of hallucinated tokens. If you see drift into nonsense, drop top-p to 0.85 before touching temperature.
Code generation (structured, syntax-constrained)
temperature: 0.1
top_p: 0.95Code wants determinism but benefits from a slightly wider nucleus — variable names, equivalent library calls, and formatting choices all have similar probability. Temperature 0.1 keeps syntax valid; top-p 0.95 admits stylistic variants.
Brainstorming / divergent thinking
temperature: 1.0
top_p: 0.9Full model entropy, but top-p still clips the absolute garbage. Use this for “give me 20 ideas” prompts where you’ll filter downstream. Never ship this directly to users.
Common pitfalls
Setting both to extreme values
temperature: 1.5
top_p: 0.95This is the “hallucination accelerator.” High temperature flattens the distribution; high top-p admits the flattened tail. You’ll get confident-sounding fabrications with perfect grammar. If you need more creativity, raise temperature or top-p, not both.
Treating top-p as a “creativity dial”
Top-p controls which tokens are eligible, not how randomly they’re chosen. A top-p of 0.5 with temperature 1.0 is not “half creative” — it’s “sample randomly from the top 50% of probability mass,” which often means sampling from a weirdly bimodal distribution. Temperature governs randomness; top-p governs eligibility.
Ignoring the interaction with repetition penalties
Repetition penalties (presence_penalty, frequency_penalty) modify logits before temperature scaling. If you use penalties, your effective temperature is higher than the configured value because penalties already flattened the distribution. Compensate by lowering temperature 0.1–0.2 when penalties are active.
Assuming defaults are safe
OpenAI defaults: temperature=1.0, top_p=1.0 (no nucleus filtering). Anthropic defaults: temperature=1.0, top_p=0.99 (effectively no filtering). These are chat defaults, not production defaults. Explicitly set both in every production call.
Tuning process for a new use case
-
Start with temperature=0.2, top_p=0.9. Run 50–100 samples. Measure your quality metric (exact match, BLEU, human eval, downstream task success).
-
If outputs are too repetitive or stuck in loops: raise top_p to 0.95. This admits more alternatives without increasing randomness.
-
If outputs are factually inconsistent: lower temperature to 0.1. Keep top_p at 0.9.
-
If outputs are too conservative/boring: raise temperature to 0.4. Keep top_p at 0.9.
-
If you see semantic drift or nonsense: lower top_p to 0.85. This is your strongest guardrail against tail tokens.
-
Lock both values. Document the combination and the evaluation dataset it was tuned against. Re-evaluate when model versions change.
Never tune both simultaneously. Change one, evaluate, then change the other.
Code: parameter validation helper
def validate_sampling_params(
temperature: float,
top_p: float,
*,
allow_deterministic: bool = True
) -> tuple[float, float]:
"""
Validate and normalize temperature and top-p.
Raises ValueError for known-bad combinations.
Returns (temperature, top_p) possibly adjusted.
"""
if not 0.0 <= temperature <= 2.0:
raise ValueError(f"temperature must be in [0, 2], got {temperature}")
if not 0.0 < top_p <= 1.0:
raise ValueError(f"top_p must be in (0, 1], got {top_p}")
# Deterministic mode: top_p is ignored by the model, but we normalize
if temperature == 0.0:
if not allow_deterministic:
raise ValueError("temperature=0 requires allow_deterministic=True")
return 0.0, 1.0
# Warn on dangerous combinations
if temperature > 1.2 and top_p > 0.95:
import warnings
warnings.warn(
f"High temperature ({temperature}) with high top_p ({top_p}) "
"often produces hallucinations. Consider lowering one.",
UserWarning,
stacklevel=2
)
# Warn on redundant combinations
if temperature < 0.3 and top_p > 0.95:
import warnings
warnings.warn(
f"Low temperature ({temperature}) with high top_p ({top_p}) "
"is redundant — nucleus filtering has no effect.",
UserWarning,
stacklevel=2
)
return temperature, top_pCode: sampling loop for evaluation
import os
from openai import OpenAI
from dataclasses import dataclass
from typing import Iterable
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
@dataclass
class SampleConfig:
temperature: float
top_p: float
n_samples: int = 10
def generate_samples(
prompt: str,
configs: list[SampleConfig],
model: str = "gpt-4o-mini"
) -> dict[SampleConfig, list[str]]:
"""
Generate samples across a grid of (temperature, top_p) configs.
Returns mapping from config to list of generated strings.
"""
results = {}
for cfg in configs:
cfg.temperature, cfg.top_p = validate_sampling_params(
cfg.temperature, cfg.top_p
)
completions = []
for _ in range(cfg.n_samples):
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=cfg.temperature,
top_p=cfg.top_p,
max_tokens=256,
)
completions.append(resp.choices[0].message.content)
results[cfg] = completions
return resultsDebugging when outputs go wrong
Symptom: “The model repeats the same phrase 5 times”
Cause: Temperature too low, nucleus too small, or both. Fix: Raise top_p to 0.95 first. If that fails, raise temperature to 0.3.
Symptom: “Output starts coherent then descends into word salad”
Cause: High temperature + high top-p admitting tail tokens that derail context. Fix: Drop top_p to 0.85. If still bad, drop temperature to 0.5.
Symptom: “JSON output has invalid syntax”
Cause: Temperature too high for structured formats. Fix: Temperature 0.0–0.1, top_p 0.95–1.0. Use a JSON schema validator in your pipeline regardless.
Symptom: “Same prompt gives wildly different answers across runs”
Cause: Temperature ≥ 1.0 with no top-p restriction. Fix: Set top_p ≤ 0.9. If variance persists, lower temperature.
What changes with model scale
Larger models have sharper native distributions — the top token often carries 40–60% of probability mass. This means:
- Low temperature has stronger effect on large models. Temperature 0.2 on a 70B model may be equivalent to 0.4 on a 7B model.
- Top-p thresholds admit fewer tokens at the same p-value. Top-p 0.9 on a large model might admit 5 tokens; on a small model, 20.
- Calibrate per model family. Don’t copy parameters from GPT-4o to Llama-3-70B without re-tuning.
When to use top-k instead
Top-k (sample from the k most likely tokens) is a hard cap; top-p is a probability-mass cap. Use top-k when:
- You need strict token budgets for latency (k=40 is common in vLLM/TGI)
- You’re implementing speculative decoding and need deterministic candidate sets
- Regulatory requirements mandate a hard limit on vocabulary exposure
Otherwise, top-p adapts to the model’s confidence; top-k does not. Most production systems use top-p exclusively.
Summary checklist
- Set both temperature and top_p explicitly in every production call
- Start with temperature=0.2, top_p=0.9 for factual tasks
- Tune one parameter at a time, evaluate after each change
- Avoid high-high combinations (temp > 1.2, top_p > 0.95)
- Validate parameters at call time with a helper like
validate_sampling_params - Re-tune when model versions change
- Log the exact parameters used for every generation — you’ll need them for debugging
The interaction between temperature and top-p together is not symmetric. Temperature shapes the distribution; top-p gates it. Master the gate first, then shape the distribution.