How temperature changes the logits before sampling

Temperature is the single most misunderstood knob in LLM inference. Engineers treat it as a “creativity dial” when it’s really a logit scaling factor that reshapes the probability distribution before sampling. If you’ve ever wondered why temperature 0.7 feels different from 0.8, or why 0 breaks greedy decoding while 2.0 produces garbage, this guide walks through the exact transformation and its consequences.

The math: temperature scales logits before softmax

The standard softmax converts logits $z_i$ to probabilities $p_i$:

$$p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$

Temperature $T$ divides every logit before that softmax:

$$p_i(T) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$$

That’s it. Division by $T$ is the entire operation. When $T < 1$, logits spread apart — the largest logit gets relatively larger, the smallest relatively smaller. When $T > 1$, logits compress toward each other. At $T \to 0$, the distribution becomes a one-hot vector at the argmax. At $T \to \infty$, it approaches uniform.

import numpy as np

def softmax(logits, temperature=1.0):
    scaled = logits / temperature
    exp = np.exp(scaled - np.max(scaled))  # numerical stability
    return exp / exp.sum()

logits = np.array([5.0, 3.0, 1.0, -1.0, -3.0])

for T in [0.1, 0.5, 1.0, 1.5, 2.0, 5.0]:
    probs = softmax(logits, T)
    print(f"T={T:.1f}: {probs.round(4)}  entropy={-(probs * np.log(probs + 1e-12)).sum():.3f}")

Output:

T=0.1: [1.0000 0.0000 0.0000 0.0000 0.0000]  entropy=0.000
T=0.5: [0.9999 0.0001 0.0000 0.0000 0.0000]  entropy=0.001
T=1.0: [0.8438 0.1142 0.0316 0.0087 0.0017]  entropy=0.608
T=1.5: [0.6724 0.1892 0.0722 0.0346 0.0116]  entropy=1.005
T=2.0: [0.5585 0.2182 0.1038 0.0592 0.0203]  entropy=1.228
T=5.0: [0.2835 0.2277 0.1829 0.1470 0.1189]  entropy=1.562

Notice how entropy increases monotonically with temperature. That’s the only thing temperature does — it controls the entropy of the output distribution.

What temperature actually changes in practice

Greedy vs. sampling boundary

At exactly $T=0$, you cannot sample — the distribution is degenerate. Most inference engines special-case this to argmax (greedy decoding). Any $T > 0$ enables sampling, even $T=0.001$. The difference between $T=0$ and $T=0.001$ is infinite in principle (deterministic vs. stochastic) but negligible in practice for the top token.

# Most engines implement this pattern
def sample_next_token(logits, temperature=1.0, top_k=None, top_p=None):
    if temperature == 0:
        return int(np.argmax(logits))  # greedy
    probs = softmax(logits, temperature)
    # ... apply top-k, top-p, then sample

The “sharpness” intuition

Low temperature makes the model “more confident” — it amplifies the model’s existing preferences. High temperature makes it “less confident” — it flattens differences the model learned. Neither is inherently more creative. A model that assigns 0.9 to “the” and 0.1 to “a” at $T=1$ will assign ~0.99 and ~0.01 at $T=0.5$. It’s not discovering new options; it’s doubling down on its top choice.

Choosing temperature: an ordered decision path

1. Start with the task category

2. Adjust for model size and training

Larger models (70B+) tend to have sharper logit distributions — their top logits are further from the rest. They often need higher temperature to achieve the same entropy as smaller models. A 7B model at $T=0.7$ may behave like a 70B model at $T=0.9$.

Instruction-tuned models are calibrated differently than base models. Base models often benefit from $T=0.7-1.0$ for coherent text. Instruction-tuned models are frequently run at $T=0.1-0.3$ for assistant-like responses.

3. Combine with top-p (nucleus sampling)

Temperature and top-p interact. Top-p truncates the tail after temperature scaling. Common combinations:

# Conservative: low temp + tight nucleus
# Good for: factual QA, code
temperature=0.2, top_p=0.9

# Balanced: moderate temp + standard nucleus  
# Good for: general chat
temperature=0.7, top_p=0.95

# Exploratory: higher temp + wide nucleus
# Good for: creative writing
temperature=1.0, top_p=0.99

Avoid $T > 1.0$ with $top_p < 0.9$ — you’re flattening the distribution then aggressively truncating it, which wastes the temperature increase.

4. Validate with entropy, not vibes

Log the entropy of your sampled distributions in production. It’s the only metric that tells you what temperature is actually doing.

def entropy(probs):
    return -(probs * np.log(probs + 1e-12)).sum()

# In your sampling loop:
probs = softmax(logits, temperature)
token = np.random.choice(len(probs), p=probs)
print(f"step entropy: {entropy(probs):.3f}")  # log this

Target entropy ranges (rough guidelines for vocabulary ~50k):

• 0.0 – 0.5: near-deterministic
• 0.5 – 1.5: focused but stochastic
• 1.5 – 3.0: diverse, coherent
• 3.0+: approaching random

Common pitfalls

Temperature 0 is not “low temperature”

$T=0$ is a special case that disables sampling entirely. If you want “mostly deterministic but with occasional variation,” use $T=0.1$ or $T=0.2$, not $T=0$. The jump from $T=0$ to $T=0.1$ is qualitatively different from $T=0.1$ to $T=0.2$.

Temperature does not fix bad logits

If your model assigns 0.99 probability to a wrong token, $T=2.0$ only reduces it to ~0.9. The model’s relative preferences are preserved — temperature only changes how sharply they’re expressed. Fix the model (prompt, fine-tune, RAG) rather than cranking temperature.

Floating point precision at low temperature

At $T < 0.1$, $\exp(z_i / T)$ overflows float32 for typical logit ranges. Always subtract max before exponentiating:

# Correct: subtract max for numerical stability
scaled = logits / temperature
exp = np.exp(scaled - np.max(scaled))
probs = exp / exp.sum()

# Wrong: will overflow at low T
exp = np.exp(logits / temperature)  # don't do this

Temperature interacts with repetition penalty

Repetition penalty modifies logits before temperature scaling. The order matters:

# Typical pipeline order
logits = model(input_ids)
logits = apply_repetition_penalty(logits, generated_tokens, penalty=1.1)
logits = apply_temperature(logits, temperature)
probs = softmax(logits)

If you apply temperature first, the penalty gets diluted. Most engines do it in the order above.

Tradeoffs to recognize

Coherence vs. diversity

This is the fundamental tradeoff. Lower temperature = more coherent, more repetitive, less surprising. Higher temperature = more diverse, more surprising, less coherent. There is no free lunch.

Determinism vs. exploration

For unit tests and regression suites, you need $T=0$ (or fixed seed + $T>0$). For generating synthetic training data, you need $T>0$. Don’t use the same temperature for both.

Latency implications

Temperature itself adds negligible compute. But higher temperature often leads to longer generations (less likely to hit EOS early), which increases total latency. If you’re latency-sensitive, this secondary effect matters.

Provider-specific behavior

Different inference providers implement temperature slightly differently. Some clamp minimum temperature to 0.01. Some apply top-k before temperature. Some use different numerical precision. If you’re switching providers, re-validate your temperature settings — don’t assume $T=0.7$ means the same thing everywhere.

Quick reference: temperature cheat sheet

# Copy-paste this into your config
TEMPERATURE_PRESETS = {
    "deterministic": 0.0,      # greedy, for tests/extraction
    "factual": 0.2,            # QA, summarization, code
    "balanced": 0.7,           # general chat default
    "creative": 1.0,           # stories, brainstorming
    "exploratory": 1.3,        # synthetic data, max diversity
}

# Pair with top-p
TOP_P_PRESETS = {
    "deterministic": 1.0,      # unused at T=0
    "factual": 0.9,
    "balanced": 0.95,
    "creative": 0.99,
    "exploratory": 0.99,
}

Debugging checklist

When generations feel wrong, check in order:

1. Log the raw logits for the first few steps. Are they reasonable? (Top logit ~5-10, gap to second ~1-3)
2. Log the post-temperature probabilities. Does the distribution look like what you intended?
3. Log entropy per step. Is it in your target range?
4. Try $T=0$. If greedy output is already bad, temperature won’t fix it.
5. Check repetition penalty. Too high (>1.2) creates artifacts that look like temperature problems.
6. Verify provider behavior. Some APIs ignore temperature when top-p=1.0 or apply undocumented clamping.

Temperature is a logit scaler. That’s the complete mental model. Everything else — creativity, randomness, hallucination rate — emerges from how that scaling reshapes the probability distribution. Set it by targeting entropy for your task, validate with logs, and move on.