Choosing a temperature for chatbots vs summarization

Choosing the right temperature for chatbots vs summarization is one of the most impactful sampling decisions you’ll make. Temperature controls the probability distribution over tokens: low values concentrate mass on high-probability continuations, while high values flatten the distribution, allowing more surprising outputs. Chatbots typically need enough randomness to feel conversational, but summarization demands fidelity to source material. This guide walks through selecting, testing, and locking in temperature values for each use case.

Step 1: Understand what temperature actually does

Temperature (τ) scales the logits before the softmax: p_i = exp(logit_i / τ) / Σ exp(logit_j / τ). At τ → 0, the model becomes deterministic (argmax). At τ = 1, you get the raw model distribution. At τ > 1, the distribution flattens, increasing entropy.

For chatbots, you want controlled creativity — enough variation to avoid repetitive loops, but not so much that the persona drifts. For summarization, you want the model to stick to the source text; hallucinations increase sharply as temperature rises above 0.3.

# Quick visualization of temperature effect on a fixed logit vector
import numpy as np

logits = np.array([5.0, 3.0, 1.0, 0.5, 0.1])  # Token A strongly preferred

def softmax_with_temp(logits, temp):
    scaled = logits / temp
    exp = np.exp(scaled - scaled.max())  # Numerical stability
    return exp / exp.sum()

for t in [0.1, 0.3, 0.7, 1.0, 1.5]:
    probs = softmax_with_temp(logits, t)
    print(f"τ={t}: {probs.round(3)}")

Run this to see how probability mass shifts. At τ=0.1, the top token gets ~99% probability. At τ=1.5, it drops to ~40%.

Step 2: Set baseline temperatures for each task

Start with these empirically grounded baselines, then adjust per Step 3.

These baselines assume top-p (nucleus sampling) is also set. A common pairing: temperature=0.7, top_p=0.9 for chat; temperature=0.2, top_p=0.9 for summarization.

# Recommended default configs
CHAT_DEFAULTS = {
    "temperature": 0.7,
    "top_p": 0.9,
    "max_tokens": 512,
    "frequency_penalty": 0.1,
    "presence_penalty": 0.1,
}

SUMMARIZATION_DEFAULTS = {
    "temperature": 0.2,
    "top_p": 0.9,
    "max_tokens": 256,
    "frequency_penalty": 0.2,  # Discourage repetition in summaries
    "presence_penalty": 0.0,
}

Step 3: Build a temperature sweep evaluation harness

Don’t guess — measure. Create a small evaluation set (20–50 examples per task) and run each at multiple temperatures. Score outputs automatically where possible, then spot-check.

import json
from dataclasses import dataclass
from typing import List, Callable
import numpy as np

@dataclass
class EvalCase:
    input_text: str
    reference: str = ""  # For summarization: source text. For chat: expected behavior description.
    task_type: str  # "chat" or "summarize"

@dataclass
class EvalResult:
    case: EvalCase
    temperature: float
    output: str
    metrics: dict

def run_temperature_sweep(
    cases: List[EvalCase],
    temperatures: List[float],
    generate_fn: Callable[[str, dict], str],
    base_params: dict,
) -> List[EvalResult]:
    """Run each case at each temperature, return results."""
    results = []
    for case in cases:
        for temp in temperatures:
            params = {**base_params, "temperature": temp}
            output = generate_fn(case.input_text, params)
            
            # Compute automatic metrics
            metrics = compute_metrics(case, output)
            
            results.append(EvalResult(
                case=case,
                temperature=temp,
                output=output,
                metrics=metrics
            ))
    return results

def compute_metrics(case: EvalCase, output: str) -> dict:
    """Task-specific automatic metrics."""
    if case.task_type == "summarize":
        return summarization_metrics(case.reference, output)
    else:
        return chat_metrics(case, output)

def summarization_metrics(source: str, summary: str) -> dict:
    """Heuristic metrics for summarization quality."""
    # Length ratio (target ~0.1-0.3 for typical summarization)
    length_ratio = len(summary.split()) / max(len(source.split()), 1)
    
    # Coverage: fraction of source sentences with at least one overlapping content word
    source_sents = source.split('. ')
    summary_words = set(summary.lower().split())
    covered = 0
    for sent in source_sents:
        sent_words = set(sent.lower().split())
        content_words = {w for w in sent_words if len(w) > 3 and w not in {'the', 'and', 'for', 'are', 'but', 'not', 'you', 'all', 'any', 'can', 'her', 'was', 'one', 'our', 'out', 'day', 'get', 'has', 'him', 'his', 'how', 'its', 'may', 'new', 'now', 'old', 'see', 'two', 'way', 'who', 'boy', 'did', 'man', 'men', 'put', 'say', 'she', 'too', 'use'}}
        if content_words & summary_words:
            covered += 1
    coverage = covered / max(len(source_sents), 1)
    
    # Repetition penalty: fraction of n-grams that appear more than once
    words = summary.lower().split()
    bigrams = [tuple(words[i:i+2]) for i in range(len(words)-1)]
    unique_bigrams = set(bigrams)
    repetition = 1 - (len(unique_bigrams) / max(len(bigrams), 1))
    
    return {
        "length_ratio": round(length_ratio, 3),
        "coverage": round(coverage, 3),
        "repetition": round(repetition, 3),
    }

def chat_metrics(case: EvalCase, output: str) -> dict:
    """Heuristic metrics for chat quality."""
    words = output.split()
    # Response length (avoid too short / too long)
    length = len(words)
    
    # Repetition
    bigrams = [tuple(words[i:i+2]) for i in range(len(words)-1)]
    unique_bigrams = set(bigrams)
    repetition = 1 - (len(unique_bigrams) / max(len(bigrams), 1))
    
    # Question detection (for conversational flow)
    has_question = '?' in output
    
    return {
        "length": length,
        "repetition": round(repetition, 3),
        "has_question": has_question,
    }

Step 4: Define your acceptance criteria

Automatic metrics are proxies. Define concrete thresholds that map to your product requirements.

For summarization, a reasonable starting rubric:

• Length ratio: 0.08–0.25 (adjust for your domain)
• Coverage: ≥ 0.6 (at least 60% of source sentences reflected)
• Repetition: ≤ 0.15 (low n-gram duplication)

For chatbots:

• Length: 15–200 tokens (adjust for your use case)
• Repetition: ≤ 0.2
• Conversational markers: Questions, acknowledgments, or follow-ups present in ≥ 70% of multi-turn exchanges

def passes_thresholds(metrics: dict, task_type: str) -> bool:
    if task_type == "summarize":
        return (
            0.08 <= metrics["length_ratio"] <= 0.25 and
            metrics["coverage"] >= 0.6 and
            metrics["repetition"] <= 0.15
        )
    else:  # chat
        return (
            15 <= metrics["length"] <= 200 and
            metrics["repetition"] <= 0.2
        )

def find_best_temperature(results: List[EvalResult]) -> dict:
    """Aggregate pass rates per temperature."""
    by_temp = {}
    for r in results:
        by_temp.setdefault(r.temperature, {"pass": 0, "total": 0})
        by_temp[r.temperature]["total"] += 1
        if passes_thresholds(r.metrics, r.case.task_type):
            by_temp[r.temperature]["pass"] += 1
    
    best = {}
    for temp, counts in by_temp.items():
        rate = counts["pass"] / counts["total"]
        best[temp] = rate
    return dict(sorted(best.items(), key=lambda x: -x[1]))

Step 5: Run human evaluation on the top 2–3 candidates

Automatic metrics correlate imperfectly with human judgment. Take the top 2–3 temperatures from Step 4 and run a blind side-by-side evaluation.

def prepare_human_eval(results: List[EvalResult], top_temps: List[float], n_samples: int = 10) -> List[dict]:
    """Create annotation-ready samples."""
    import random
    filtered = [r for r in results if r.temperature in top_temps]
    random.shuffle(filtered)
    samples = filtered[:n_samples]
    
    annotation_tasks = []
    for r in samples:
        annotation_tasks.append({
            "input": r.case.input_text,
            "output": r.output,
            "temperature": r.temperature,
            "task_type": r.case.task_type,
            "metrics": r.metrics,
        })
    return annotation_tasks

Annotation guidelines for summarization:

1. Faithfulness: Does the summary contradict the source? (1–5)
2. Completeness: Are the main points covered? (1–5)
3. Conciseness: Is it free of fluff? (1–5)
4. Fluency: Does it read naturally? (1–5)

Annotation guidelines for chat:

1. Relevance: Does it address the user? (1–5)
2. Persona consistency: Does it stay in character? (1–5)
3. Engagement: Does it invite continuation? (1–5)
4. Safety: Any harmful content? (binary)

Collect 3+ annotations per sample. Choose the temperature with the highest mean composite score.

Step 6: Lock in the temperature and add guardrails

Once you’ve selected a temperature, hard-code it in your production path. Don’t expose it as a user-facing control unless you have a specific reason.

# Production config - temperature is fixed per endpoint
PRODUCTION_CONFIGS = {
    "chat": {
        "model": "gpt-4o-mini",
        "temperature": 0.7,
        "top_p": 0.9,
        "max_tokens": 512,
        "frequency_penalty": 0.1,
        "presence_penalty": 0.1,
        "stop": ["User:", "Human:"],  # Prevent role confusion
    },
    "summarize": {
        "model": "gpt-4o-mini",
        "temperature": 0.2,
        "top_p": 0.9,
        "max_tokens": 256,
        "frequency_penalty": 0.2,
        "presence_penalty": 0.0,
        "stop": ["\n\nSource:", "\n\nOriginal:"],  # Prevent leakage
    },
}

def call_llm(endpoint: str, messages: list, config: dict) -> str:
    """Wrapper that enforces production config."""
    import openai
    client = openai.OpenAI()
    
    # Merge with any runtime overrides (e.g., max_tokens for long contexts)
    params = {**config}
    params["messages"] = messages
    
    response = client.chat.completions.create(**params)
    return response.choices[0].message.content

Add runtime guardrails that catch temperature-sensitive failure modes:

def validate_output(output: str, task_type: str, source_text: str = "") -> tuple[bool, str]:
    """Post-generation validation. Returns (pass, reason)."""
    if task_type == "summarize":
        # Hallucination check: flag entities in output not in source
        import re
        source_entities = set(re.findall(r'\b[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*\b', source_text))
        output_entities = set(re.findall(r'\b[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*\b', output))
        hallucinated = output_entities - source_entities
        if len(hallucinated) > 2:
            return False, f"Potential hallucinated entities: {hallucinated}"
        
        # Length check
        ratio = len(output.split()) / max(len(source_text.split()), 1)
        if ratio > 0.4:
            return False, f"Summary too long (ratio={ratio:.2f})"
    
    elif task_type == "chat":
        # Repetition check
        words = output.lower().split()
        if len(words) > 10:
            trigrams = [tuple(words[i:i+3]) for i in range(len(words)-2)]
            unique = set(trigrams)
            if len(unique) / len(trigrams) < 0.5:
                return False, "Excessive repetition detected"
    
    return True, "OK"

Step 7: Monitor temperature-sensitive metrics in production

Temperature choice isn’t a one-time decision. Model updates, prompt changes, and distribution shift can all degrade quality. Track these metrics per endpoint:

# Example metrics to emit (adapt to your observability stack)
PRODUCTION_METRICS = {
    "chat": [
        "avg_response_length_tokens",
        "repetition_rate_3gram",
        "user_turn_count_per_session",
        "explicit_feedback_positive_rate",
        "safety_filter_trigger_rate",
    ],
    "summarize": [
        "avg_length_ratio",
        "entity_hallucination_rate",  # Requires NER on source + output
        "coverage_estimate",  # Heuristic: source sentence overlap
        "user_edit_rate",  # If users edit the summary
        "regeneration_rate",  # User asks for new summary
    ],
}

Set alerts on:

• Repetition rate > 0.25 for chat (indicates temperature too low or context stuffing)
• Hallucination rate > 0.05 for summarization (temperature may be too high, or prompt needs tightening)
• Regeneration rate > 0.3 (users dissatisfied, revisit temperature or prompt)

Step 8: Document the decision and revisit quarterly

Record the evaluation data, chosen temperature, and rationale in your model card or config repo. Include:

• Evaluation dataset description and size
• Automatic metric scores per temperature tested
• Human evaluation results (mean scores, inter-annotator agreement)
• Production guardrails in place
• Date of last review

# Temperature Decision Record: Chat Endpoint

**Selected temperature**: 0.7
**Date**: 2024-11-15
**Model**: gpt-4o-mini (2024-07-18)

## Evaluation summary
| Temperature | Auto pass rate | Human composite (1-5) | Notes |
|-------------|----------------|----------------------|-------|
| 0.3 | 0.92 | 3.1 | Too robotic, low engagement |
| 0.5 | 0.87 | 3.8 | Good balance |
| **0.7** | **0.79** | **4.2** | **Best engagement, acceptable coherence** |
| 0.9 | 0.61 | 3.9 | More creative but higher repetition |
| 1.0 | 0.44 | 3.2 | Persona drift observed |

## Human eval details
- 3 annotators, 30 samples per temperature
- Krippendorff's α = 0.72 (acceptable agreement)
- Composite = mean(faithfulness, engagement, safety)

## Guardrails
- Repetition filter: 3-gram uniqueness < 0.5 → regenerate at τ=0.5
- Safety filter: Perspective API > 0.7 → block + fallback

## Next review
Scheduled: 2025-02-15 or on model version change.

Verification checklist

After deploying your chosen temperature, verify success with this checklist:

• Automatic metrics stable: Run the Step 3 harness against production traffic samples weekly. Pass rates should stay within ±5% of evaluation baseline.
• Human spot-check passes: Sample 20 outputs per week, have a domain expert rate them. Composite score should not drop below your acceptance threshold.
• Guardrails firing at expected rate: Repetition filter triggers on 1–3% of chat responses; hallucination filter triggers on <1% of summaries. If rates spike, investigate prompt or model changes first.
• No temperature leakage: Confirm the production endpoint ignores any temperature field sent by clients. Log rejected overrides.
• Fallback behavior tested: If your gateway supports automatic fallback (e.g., n4n.ai routes to a backup provider when the primary is degraded), verify the fallback model produces acceptable output at the same temperature. Different models calibrate temperature differently — a τ=0.7 on one model may behave like τ=0.5 on another.

Common pitfalls to avoid

Using the same temperature for everything. Chat and summarization have fundamentally different entropy requirements. A single default (often 0.7 or 1.0) hurts both tasks.

Treating temperature as a creativity dial only. It also controls consistency, hallucination rate, and repetition. Low temperature isn’t “less creative” — it’s “more deterministic.”

Ignoring top-p interaction. Temperature and top-p compound. If you lower temperature but keep top_p=1.0, you still sample from the full vocabulary. Pair them intentionally.

Skipping human evaluation. Automatic metrics for summarization (ROUGE, BERTScore) correlate poorly with faithfulness. For chat, they correlate poorly with engagement. Budget for annotation.

Setting temperature once and forgetting it. Model providers update models silently. A temperature that worked in January may hallucinate in June. Schedule quarterly re-evaluation.

Temperature for chatbots vs summarization isn’t a single number — it’s a decision process. Run the sweep, measure against your actual requirements, involve human judgment, then lock it down with monitoring. The 30 minutes you spend on Steps 3–5 will save weeks of debugging subtle quality regressions later.