Best top-p value for factual, low-variance answers

Finding the best top-p value for factual answers isn’t a one-time guess — it’s a calibration problem. Most teams pick 0.9 because it’s the default, then wonder why their extraction pipeline hallucinates entity names or drifts on numeric values. The right setting depends on your model, your prompt structure, and whether you’re optimizing for exact-match accuracy or controlled diversity. This guide gives you a repeatable process to dial it in, with code you can drop into your evaluation harness.

Step 1: Understand what top-p actually controls

Top-p (nucleus sampling) truncates the probability distribution at a cumulative threshold p, then renormalizes and samples from the remaining mass. At p=1.0 you get the full distribution (pure sampling). At p→0 you approach greedy decoding (argmax). The parameter trades off diversity against fidelity to the model’s highest-probability tokens.

For factual workloads — entity extraction, SQL generation, classification, RAG answer synthesis — you want the model to stay on the highest-probability reasoning path. That means low top-p. But setting it too low (0.01–0.05) can cause repetition loops or force the model into locally optimal but globally wrong completions, especially on longer generations.

The best top-p value for factual answers typically lands between 0.1 and 0.3 for most instruction-tuned models. Your job is to find the exact number for your stack.

Step 2: Build a minimal evaluation harness

Don’t tune by vibes. Build a small, representative dataset (50–200 examples) that covers your failure modes: ambiguous entities, numeric precision, multi-hop reasoning, and known hallucination triggers. Each example needs a prompt template and a ground-truth answer or validation function.

# eval_harness.py
import json
import asyncio
from dataclasses import dataclass
from typing import Callable, Awaitable
from openai import AsyncOpenAI

@dataclass
class EvalCase:
    name: str
    prompt: str
    expected: str | None = None
    validate: Callable[[str], bool] | None = None

async def run_eval(
    client: AsyncOpenAI,
    model: str,
    cases: list[EvalCase],
    top_p: float,
    temperature: float = 0.0,
    max_tokens: int = 512,
) -> dict[str, float]:
    """Run cases at a single top-p setting. Returns pass rate and latency stats."""
    results = []
    latencies = []
    
    for case in cases:
        start = asyncio.get_event_loop().time()
        resp = await client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": case.prompt}],
            top_p=top_p,
            temperature=temperature,
            max_tokens=max_tokens,
        )
        latency = asyncio.get_event_loop().time() - start
        latencies.append(latency)
        
        output = resp.choices[0].message.content or ""
        
        if case.validate:
            passed = case.validate(output)
        elif case.expected:
            passed = output.strip() == case.expected.strip()
        else:
            passed = False
        
        results.append(passed)
    
    return {
        "pass_rate": sum(results) / len(results),
        "avg_latency_ms": sum(latencies) / len(latencies) * 1000,
        "total_cases": len(cases),
    }

Keep temperature at 0.0 while tuning top-p. Temperature and top-p interact non-linearly; isolate one variable at a time.

Step 3: Define your factual test cases

Your test cases should reflect real failure modes, not synthetic benchmarks. Here are patterns that expose top-p sensitivity:

# test_cases.py
from eval_harness import EvalCase
import re

def validate_sql(output: str) -> bool:
    """Check for valid SELECT syntax and no hallucinated tables."""
    output = output.strip()
    if not output.upper().startswith("SELECT"):
        return False
    # Reject common hallucinated table names
    banned = ["users", "orders", "products", "customers"]  # adjust to your schema
    return not any(b in output.lower() for b in banned)

def validate_entity_extraction(output: str) -> bool:
    """Exact match on entity list, order-independent."""
    expected = {"Acme Corp", "John Smith", "2024-01-15", "Invoice #4421"}
    found = set(re.findall(r'\b\w+(?:[\s#-]\w+)*\b', output))
    return expected.issubset(found)

def validate_numeric(output: str) -> bool:
    """Extract first number, check within 1% of expected."""
    match = re.search(r'[\d,]+\.?\d*', output.replace(',', ''))
    if not match:
        return False
    try:
        val = float(match.group())
        return 99.0 <= val <= 101.0  # expected ~100
    except ValueError:
        return False

CASES = [
    EvalCase(
        name="sql_generation",
        prompt="""Schema: transactions(id, amount, merchant_id, created_at), merchants(id, name, category).
Write SQL for: total amount per merchant category in January 2024.""",
        validate=validate_sql,
    ),
    EvalCase(
        name="entity_extraction",
        prompt="""Extract all entities from: "Acme Corp paid John Smith $12,500 on 2024-01-15 per Invoice #4421."
Return as JSON list.""",
        validate=validate_entity_extraction,
    ),
    EvalCase(
        name="numeric_precision",
        prompt="""Calculate: (150 * 0.67) + (200 * 0.33). Return only the number.""",
        validate=validate_numeric,
    ),
    EvalCase(
        name="classification",
        prompt="""Classify: "The server returned 503 for 45 minutes yesterday."
Categories: [incident, maintenance, deployment, false_alarm].
Return only the category.""",
        expected="incident",
    ),
]

Step 4: Sweep top-p systematically

Run a grid search across the plausible range. Log pass rate, latency, and qualitative samples at each setting.

# sweep.py
import asyncio
import csv
from eval_harness import run_eval
from test_cases import CASES
from openai import AsyncOpenAI

async def main():
    client = AsyncOpenAI()  # or your gateway endpoint
    model = "gpt-4o-mini"   # or your deployed model
    
    top_p_values = [0.0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5, 0.7, 0.9, 1.0]
    
    with open("top_p_sweep.csv", "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["top_p", "pass_rate", "avg_latency_ms", "case", "passed"])
        
        for top_p in top_p_values:
            result = await run_eval(client, model, CASES, top_p=top_p)
            print(f"top_p={top_p:.2f} pass_rate={result['pass_rate']:.2%} latency={result['avg_latency_ms']:.0f}ms")
            
            # Re-run with per-case detail for CSV
            for case in CASES:
                detail = await run_eval(client, model, [case], top_p=top_p)
                writer.writerow([top_p, detail["pass_rate"], detail["avg_latency_ms"], case.name, detail["pass_rate"]])

if __name__ == "__main__":
    asyncio.run(main())

Run this overnight or in CI. You’ll get a CSV you can plot. Look for the plateau — the highest pass rate before latency or repetition degrades.

Step 5: Inspect qualitative failures at each setting

Numbers lie. At each top-p value, capture 3–5 completions per case and read them. You’re looking for:

• Repetition loops (top-p too low): “The answer is 100. The answer is 100. The answer is 100.”
• Premature termination (top-p too low): Model cuts off mid-sentence because the next token fell below threshold
• Hallucinated entities (top-p too high): Invented table names, wrong dates, plausible-sounding but wrong numbers
• Format drift (top-p too high): JSON becomes JSONL, SQL gets markdown fences, classification returns sentences

# qualitative.py
import asyncio
from openai import AsyncOpenAI

async def sample_completions(client, model, prompt, top_p, n=5):
    resp = await client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        top_p=top_p,
        temperature=0.0,
        max_tokens=256,
        n=n,
    )
    return [c.message.content for c in resp.choices]

async def main():
    client = AsyncOpenAI()
    model = "gpt-4o-mini"
    
    prompt = """Extract all entities from: "Acme Corp paid John Smith $12,500 on 2024-01-15 per Invoice #4421."
Return as JSON list."""
    
    for top_p in [0.0, 0.1, 0.2, 0.3, 0.5, 0.9]:
        print(f"\n=== top_p={top_p} ===")
        completions = await sample_completions(client, model, prompt, top_p, n=3)
        for i, c in enumerate(completions):
            print(f"  [{i}] {c[:200]}")

asyncio.run(main())

Step 6: Choose your operating point and add guardrails

Pick the highest top-p that maintains your target pass rate. This gives you maximum diversity margin for edge cases without sacrificing accuracy. For most factual workloads on GPT-4o-class models, that’s 0.1–0.2. On smaller models (7B–13B), you may need 0.05–0.15.

Then add runtime guardrails so a bad deployment doesn’t silently degrade:

# production_wrapper.py
from openai import AsyncOpenAI
from pydantic import BaseModel, Field
from typing import Literal

class FactualResponse(BaseModel):
    content: str
    top_p_used: float
    model: str
    validation_passed: bool
    fallback_triggered: bool = False

class FactualClient:
    def __init__(
        self,
        client: AsyncOpenAI,
        model: str,
        primary_top_p: float = 0.15,
        fallback_top_p: float = 0.05,
        max_retries: int = 2,
    ):
        self.client = client
        self.model = model
        self.primary_top_p = primary_top_p
        self.fallback_top_p = fallback_top_p
        self.max_retries = max_retries
    
    async def complete(self, prompt: str, validator) -> FactualResponse:
        # Primary attempt
        for attempt in range(self.max_retries):
            resp = await self.client.chat.completions.create(
                model=self.model,
                messages=[{"role": "user", "content": prompt}],
                top_p=self.primary_top_p,
                temperature=0.0,
                max_tokens=512,
            )
            output = resp.choices[0].message.content or ""
            if validator(output):
                return FactualResponse(
                    content=output,
                    top_p_used=self.primary_top_p,
                    model=self.model,
                    validation_passed=True,
                )
        
        # Fallback to more deterministic setting
        resp = await self.client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            top_p=self.fallback_top_p,
            temperature=0.0,
            max_tokens=512,
        )
        output = resp.choices[0].message.content or ""
        return FactualResponse(
            content=output,
            top_p_used=self.fallback_top_p,
            model=self.model,
            validation_passed=validator(output),
            fallback_triggered=True,
        )

This pattern — primary at your tuned top-p, fallback at a lower value — handles the long tail of ambiguous prompts without manual intervention.

Step 7: Verify in production with shadow evaluation

Don’t stop at pre-deployment testing. Run a shadow evaluation on live traffic for at least one week before making the new top-p your default.

# shadow_eval.py
import asyncio
import random
from production_wrapper import FactualClient, FactualResponse
from test_cases import CASES, validate_sql, validate_entity_extraction, validate_numeric

VALIDATORS = {
    "sql_generation": validate_sql,
    "entity_extraction": validate_entity_extraction,
    "numeric_precision": validate_numeric,
    "classification": lambda o: o.strip() == "incident",
}

async def shadow_evaluate(client: FactualClient, n_samples: int = 1000):
    """Compare primary vs fallback top-p on production-like prompts."""
    primary_wins = 0
    fallback_wins = 0
    both_fail = 0
    
    for _ in range(n_samples):
        case = random.choice(CASES)
        validator = VALIDATORS[case.name]
        
        # Run with primary top-p
        primary_result = await client.complete(case.prompt, validator)
        
        # Run with fallback top-p (simulated by temporary override)
        fallback_client = FactualClient(
            client.client, client.model,
            primary_top_p=client.fallback_top_p,
            fallback_top_p=client.fallback_top_p,
        )
        fallback_result = await fallback_client.complete(case.prompt, validator)
        
        if primary_result.validation_passed and not fallback_result.validation_passed:
            primary_wins += 1
        elif fallback_result.validation_passed and not primary_result.validation_passed:
            fallback_wins += 1
        elif not primary_result.validation_passed and not fallback_result.validation_passed:
            both_fail += 1
    
    print(f"Primary wins: {primary_wins}")
    print(f"Fallback wins: {fallback_wins}")
    print(f"Both fail: {both_fail}")
    print(f"Primary pass rate: {primary_wins / n_samples:.2%}")
    print(f"Fallback pass rate: {fallback_wins / n_samples:.2%}")

# Run in background on staging traffic
# asyncio.run(shadow_evaluate(production_client))

If fallback wins more than 5% of cases, your primary top-p is too high. If both fail on the same cases, you have a prompt or model problem, not a sampling problem.

Step 8: Document the decision and set a re-evaluation trigger

Commit your chosen top-p to config with a comment explaining the evidence:

# config/sampling.yaml
factual_workloads:
  top_p: 0.15
  temperature: 0.0
  fallback_top_p: 0.05
  # Chosen 2024-01-20 after sweep on 150 eval cases:
  # top_p=0.15 -> 94.7% pass rate
  # top_p=0.20 -> 92.0% pass rate (hallucination increase on entity extraction)
  # top_p=0.10 -> 93.3% pass rate (repetition loops on SQL generation)
  # Re-evaluate when: model version changes, prompt template changes, or pass rate drops below 90%

Set a calendar reminder or CI check to re-run the sweep quarterly or on model upgrades. The best top-p value for factual answers drifts as models update — what worked on GPT-4o-mini-2024-07 may fail on the November refresh.

Common pitfalls to avoid

Don’t copy-paste top-p from chat use cases. Creative writing wants 0.9. Code generation often works at 0.1–0.2. Factual extraction wants 0.05–0.2. Each workload needs its own calibration.

Don’t ignore temperature interaction. If you must use temperature > 0 (e.g., for diverse few-shot examples), re-run the sweep at that temperature. The optimal top-p shifts.

Don’t treat top-p as a hallucination cure. If your prompt is ambiguous or your RAG context is noisy, no sampling parameter fixes it. Fix the prompt, fix the retrieval, then tune sampling.

Don’t skip the fallback. Even a well-tuned top-p will hit edge cases. The fallback path costs one extra API call on <5% of requests and prevents silent failures.

Verification checklist

Before declaring victory, confirm:

• Sweep covers 0.0 to 1.0 in ≤0.1 increments around the optimum
• Test cases include your actual failure modes, not just happy paths
• Qualitative inspection shows no repetition loops at chosen top-p
• Fallback path tested and logged separately in production
• Shadow evaluation runs ≥1 week with ≥1000 samples
• Config documented with evidence and re-evaluation trigger
• Alerting on validation_passed rate dropping below threshold

The best top-p value for factual answers is the one you measured, not the one you guessed. Run the sweep, read the outputs, set the fallback, and put a calendar invite on the re-evaluation. That’s the engineering approach.