Why the same prompt can give different answers twice

You send the same prompt to an LLM twice and get two different answers. This isn’t a bug — it’s the default behavior of probabilistic token generation. Understanding why it happens, which knobs actually control it, and where determinism breaks down anyway is essential for building reliable LLM-powered systems.

The core mechanism: probabilistic sampling

At inference time, an LLM outputs a probability distribution over its vocabulary for the next token. The sampling strategy determines how you pick from that distribution.

# Simplified greedy vs. temperature sampling
def greedy(logits):
    return logits.argmax(dim=-1)

def temperature_sample(logits, temperature=0.7):
    probs = torch.softmax(logits / temperature, dim=-1)
    return torch.multinomial(probs, num_samples=1)

With temperature=0 (or greedy decoding), you always pick the highest-probability token. The output becomes deterministic for a given model state. Any non-zero temperature introduces randomness — by design. This is the most common source of “same prompt, different answer.”

But temperature isn’t the only source. Even at temperature 0, you can see variance.

Seeds: what they control and what they don’t

Most APIs accept a seed parameter. The mental model: same seed + same prompt + same model = same output. Reality is messier.

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Write a haiku about debugging"}],
  "temperature": 0,
  "seed": 42
}

OpenAI documents that seed makes a best effort at determinism. Anthropic’s seed parameter behaves similarly. But the guarantee holds only when:

1. The model weights haven’t changed
2. The inference infrastructure hasn’t changed
3. No non-deterministic ops execute in the forward pass

Providers reserve the right to update model weights behind the same model identifier. A “gpt-4o” call today may run different weights than last week. The seed only controls the random number generator for sampling — it doesn’t version the model.

Floating-point non-determinism in the forward pass

Even with temperature 0 and a fixed seed, the forward pass itself can produce different logits across runs. This surprises many engineers.

Modern transformer inference uses parallel operations (matrix multiplies, attention kernels) that accumulate floating-point values in non-deterministic order. GPU kernels — especially those using tensor cores or flash attention — may reduce partial sums in different orders depending on:

• GPU architecture and driver version
• Kernel selection heuristics (which vary by input length, batch size, available shared memory)
• Concurrent workloads affecting SM occupancy

# This can produce different results on the same GPU across runs
# due to non-deterministic reduction order in attention
def scaled_dot_product_attention(q, k, v, is_causal=True):
    # Flash attention / memory-efficient attention kernels
    # use non-deterministic parallel reductions
    return F.scaled_dot_product_attention(q, k, v, is_causal=is_causal)

PyTorch documents this explicitly: torch.backends.cudnn.deterministic = True and torch.use_deterministic_algorithms(True) reduce but don’t eliminate variance, and they come with significant performance penalties (often 2-10x slower). Most inference engines (vLLM, TGI, TensorRT-LLM) prioritize throughput over bitwise determinism.

Model versioning: the silent changer

Providers version models independently of the API identifier you call. OpenAI’s gpt-4o has had multiple silent updates. Anthropic’s claude-3-5-sonnet-20241022 is explicitly versioned, but the unversioned alias claude-3-5-sonnet-latest points to whatever is current.

# You cannot pin to a specific weight snapshot via the API
# This request runs whatever "gpt-4o" means today
curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_KEY" \
  -d '{"model": "gpt-4o", "messages": [...], "temperature": 0, "seed": 42}'

If reproducibility matters, you must either:

• Use explicitly versioned model identifiers where available (e.g., gpt-4o-2024-08-06)
• Accept that the model itself may drift
• Run your own weights on controlled infrastructure

Quantization and compilation artifacts

Self-hosted models introduce additional variance sources. The same weights quantized differently (AWQ vs GPTQ vs GGUF, different group sizes, different calibration datasets) produce different outputs. Even the same quantization method with different compilation targets (TensorRT-LLM vs vLLM vs llama.cpp) can diverge due to:

• Kernel fusion differences
• FP8 vs BF16 accumulation choices
• KV cache quantization schemes

# These three commands with "the same model" will produce different outputs
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --quantization awq
tensorrt_llm build --checkpoint_dir ./llama-3.1-8b --dtype fp8
llama-server -m llama-3.1-8b-instruct-q4_k_m.gguf

If you need reproducibility across deployments, you must freeze the entire artifact: weights + quantization config + inference engine version + compilation flags.

Structured output and constrained decoding

Constrained decoding (JSON mode, regex-guided generation, grammar-based sampling) adds another layer. The constraint machinery — typically a finite-state machine or pushdown automaton guiding token selection — interacts with the sampler.

# Simplified constrained decoding loop
def constrained_generate(logits, fsm_state, allowed_tokens):
    masked_logits = logits.clone()
    masked_logits[:, ~allowed_tokens] = -float('inf')
    return sample(masked_logits), fsm_state.transition(sampled_token)

At temperature 0 with a fixed seed, constrained decoding is deterministic for a given model state. But the constraint evaluation order (especially with parallel beam search or speculative decoding) can introduce variance. Speculative decoding — where a draft model proposes tokens verified by the target model — is inherently non-deterministic because acceptance/rejection depends on the target model’s probabilities at verification time.

Practical reproducibility strategies

Given all these variance sources, here’s what actually works in production:

1. Accept temperature > 0 for quality, design for variance

Most production systems need temperature 0.3-0.7 for reasonable output quality. Build your system assuming non-determinism:

# Pattern: generate multiple candidates, select best
async def generate_with_consensus(prompt, n=3, temperature=0.7):
    candidates = await asyncio.gather(*[
        llm.complete(prompt, temperature=temperature) for _ in range(n)
    ])
    return select_best(candidates)  # Your selection logic

2. Use temperature 0 + seed for deterministic workflows

When you need bitwise reproducibility (evals, regression tests, legal/compliance):

DETERMINISTIC_CONFIG = {
    "temperature": 0,
    "seed": 42,
    "top_p": 1.0,  # Disable nucleus sampling
    "max_tokens": 512,
}

# Pin model version explicitly
response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",  # Not "gpt-4o"
    messages=[...],
    **DETERMINISTIC_CONFIG
)

3. Cache aggressively at the application layer

The most reliable determinism comes from not calling the model twice for the same input. Implement semantic caching:

import hashlib
import redis

CACHE = redis.Redis(decode_responses=True)

def cache_key(prompt, config):
    # Include all params that affect output
    blob = json.dumps({"prompt": prompt, **config}, sort_keys=True)
    return f"llm:{hashlib.sha256(blob.encode()).hexdigest()[:16]}"

async def cached_complete(prompt, config):
    key = cache_key(prompt, config)
    if cached := CACHE.get(key):
        return json.loads(cached)
    
    result = await llm.complete(prompt, **config)
    CACHE.setex(key, 86400, json.dumps(result))
    return result

4. For self-hosted: freeze the entire stack

# Dockerfile.lock - fully pinned
FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04

# Pin Python, PyTorch, inference engine
RUN pip install --no-cache-dir \
    torch==2.4.0 \
    vllm==0.6.3 \
    transformers==4.44.0

# Copy quantized model artifact (not the HF repo ID)
COPY ./models/llama-3.1-8b-instruct-awq /model

ENTRYPOINT ["vllm", "serve", "/model", "--enforce-eager"]

Build once, deploy the image everywhere. No latest tags, no dynamic quantization at startup.

5. Test for determinism, don’t assume it

Add a regression test that fails when output drifts:

import pytest

GOLDEN_PROMPTS = [
    ("Summarize this in one sentence: ...", "expected_summary"),
    ("Extract JSON: ...", '{"key": "value"}'),
]

@pytest.mark.parametrize("prompt,expected", GOLDEN_PROMPTS)
def test_deterministic_output(prompt, expected):
    # Run 3 times with same seed
    outputs = [
        llm.complete(prompt, temperature=0, seed=42)
        for _ in range(3)
    ]
    
    # All runs must match each other
    assert all(o == outputs[0] for o in outputs), "Non-deterministic across runs"
    
    # Output must match golden (update golden when model version changes)
    assert outputs[0] == expected, f"Output drifted: {outputs[0]}"

Run this in CI against your pinned model version. When the provider updates the model, the test fails — alerting you to re-evaluate.

The tradeoff table

* “High” assumes provider doesn’t silently update weights. No API provider guarantees this contractually.

What about logprobs?

Some APIs return token logprobs. You might think: “I’ll use logprobs to verify determinism or reconstruct the distribution.” Two caveats:

1. Logprobs are typically computed after sampling, from the same forward pass. They reflect the distribution that produced the sampled token — but if the forward pass is non-deterministic, the logprobs vary too.
2. Most APIs return only top-k logprobs (often k=5 or k=20). The tail of the distribution is invisible.

{
  "choices": [{
    "logprobs": {
      "content": [{
        "token": "The",
        "logprob": -0.02,
        "top_logprobs": [
          {"token": "The", "logprob": -0.02},
          {"token": "A", "logprob": -3.1}
        ]
      }]
    }
  }]
}

Logprobs are useful for confidence scoring and debugging. They don’t solve determinism.

The decisive takeaway

Same prompt, different answer is the default. Determinism requires: temperature 0, explicit seed, explicitly versioned model identifier, and a provider that doesn’t silently update weights — or self-hosted infrastructure with the entire stack frozen (weights, quantization, engine, kernel compilation).

Most production systems should not chase perfect determinism. Instead: design for variance (consensus, caching, idempotency keys), pin model versions where the API allows it, and maintain golden-set evals that catch drift when providers update models. The engineering effort to achieve true bitwise reproducibility across provider APIs exceeds the value in nearly all cases — except regulated workflows, regression testing, and offline evaluation.

If you need reproducibility for evals: use temperature 0, seed, versioned model IDs, and run your golden tests frequently. If you need it for production traffic: cache at the application layer and accept that the model itself is a moving target.