Why exact reproducibility is hard to get from an LLM API

If you have ever rerun the same prompt with temperature=0 and gotten different output, you have encountered the fundamental problem: no major LLM API offers an llm reproducibility guarantee. The industry treats determinism as a best-effort property, not a contract. This post explains why that is unlikely to change, where the variance actually comes from, and what you can do about it in production systems.

What reproducibility means for an LLM

When engineers ask for reproducibility, they usually mean bitwise identical output given identical inputs: same prompt, same parameters, same model version. In traditional software this is table stakes. In LLM inference it is structurally difficult.

The request path looks like this:

client → gateway → load balancer → model server (vLLM/TGI/etc.) → GPU kernels → logits → sampler → tokens

At every stage, implementation choices introduce variance that no seed parameter can fully control. The sampler is the only stage where a seed applies, and even there the guarantee is weaker than most people assume.

The seed parameter is not a global seed

Setting seed=42 in an OpenAI-compatible request only seeds the final multinomial sampler. It does not seed:

• CUDA kernel launch order across SMs
• FlashAttention block scheduling
• KV cache eviction or quantization noise
• Batch composition when your request shares a forward pass with others
• Model weight loading order (affects memory layout, affects numerics)

# This does NOT guarantee identical output across calls
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a haiku about Kubernetes"}],
    temperature=0,
    seed=42,
    max_tokens=50
)

Run this twice on the same endpoint. You will often get the same text, but not always. The seed only controls the RNG that draws from the probability distribution after the logits are computed. If the logits differ by even 1e-6 due to upstream non-determinism, the argmax can flip.

Hardware non-determinism is the floor

Modern GPU kernels are not deterministic by default. NVIDIA’s Tensor Cores accumulate FP16/BF16 products in FP32, but the order of accumulation depends on warp scheduling, which depends on occupancy, which depends on what else is running on the GPU.

# PyTorch demonstration of the problem
import torch

torch.manual_seed(0)
x = torch.randn(1, 4096, device="cuda", dtype=torch.bfloat16)
w = torch.randn(4096, 4096, device="cuda", dtype=torch.bfloat16)

# Two identical matmuls can produce different last-bit results
out1 = x @ w
out2 = x @ w
print((out1 != out2).any().item())  # Often True

FlashAttention compounds this. It tiles the attention computation across blocks. The reduction order within each tile depends on block scheduling. Different runs, different block orders, different rounding error accumulation. The logits drift.

Providers can enable deterministic algorithms (torch.use_deterministic_algorithms(True) or cuDNN deterministic flags), but this typically costs 15-30% throughput. No public API exposes this as a customer toggle because the economics don’t work at scale.

Batching breaks isolation

You are rarely the only request on a GPU. Inference servers batch requests dynamically to maximize utilization. Your prompt shares a forward pass with other users’ prompts.

# Simplified vLLM continuous batching concept
# Your request enters a batch with others
batch = [
    ("user_123", "Summarize this contract...", 512),
    ("user_456", "Write a poem...", 128),
    ("you", "Explain RAG...", 256),
]
# Single forward pass for all three
logits = model(batch.input_ids)  # Shared KV cache, shared compute

The presence of other sequences changes:

• KV cache memory layout and fragmentation
• PagedAttention block allocation order
• Prefill vs. decode phase timing
• Numerical precision in fused kernels (batch size affects tile sizes)

Even with temperature=0, the logits for your sequence depend on the entire batch composition. You cannot control who else is in your batch.

Model updates without version bumps

This is the most frustrating source of drift. Providers update model weights, tokenizers, or system prompts without incrementing the model identifier you see in the API.

// What you request
{ "model": "gpt-4o-mini" }

// What you might get on Tuesday vs. Friday
// - Different tokenizer merge rules (byte-level BPE changes)
// - Different system prompt injection
// - Different LoRA adapters applied server-side
// - Quantization calibration updated (INT4 → INT4 with new scales)
// - Speculative decoding draft model swapped

OpenAI’s gpt-4o-mini-2024-07-18 style version pinning helps, but not all providers offer dated snapshots. Even when they do, the dated model may still route to different hardware generations (H100 vs A100) with different numerical behavior.

Quantization and KV cache compression

Most production serving stacks quantize weights to INT4 or INT8 and compress KV caches to FP8 or INT8. The calibration statistics (scale/zero-point) are often computed per-deployment or even per-batch.

# Pseudocode for dynamic KV cache quantization
def quantize_kv_cache(kv, calibration_method="per_token"):
    if calibration_method == "per_token":
        scale = kv.abs().max(dim=-1, keepdim=True).values / 127
    elif calibration_method == "per_layer":
        scale = running_estimate_of_max(kv)
    return (kv / scale).round().to(torch.int8), scale

If the calibration method changes, or if the running estimate drifts because the batch distribution shifted, your logits shift. This is invisible to you. The API returns tokens; it does not return the quantization config that produced them.

Speculative decoding adds another layer

Many endpoints now use speculative decoding: a small draft model proposes tokens, a large target model verifies them. The acceptance/rejection decision depends on the target model’s logits at that moment.

# Speculative decoding loop (simplified)
draft_tokens = draft_model.generate(prefix, gamma=4)
target_logits = target_model(prefix + draft_tokens)
acceptance_mask = verify(draft_tokens, target_logits)
# If verification fails partway, we fall back to target model tokens

The number of accepted tokens varies with target model logits. If the target model’s numerics drift (see hardware section above), the acceptance boundary moves. Your output length and content change even though the “model” is the same.

Provider routing and fallback

Gateways that route across multiple providers or model variants introduce a reproducibility surface you cannot see.

# Hypothetical routing config
routing:
  primary: "provider_a::gpt-4o-mini"
  fallback:
    - "provider_b::gpt-4o-mini"
    - "provider_c::claude-3-haiku"
  criteria:
    - latency_p99 < 2000ms
    - error_rate < 0.01

Your request might hit provider A on Monday and provider B on Tuesday after a brief degradation. Different providers run different serving stacks (vLLM vs TGI vs TensorRT-LLM), different hardware, different quantization. The model name is the same; the inference pipeline is not.

At n4n.ai we forward provider cache-control hints and honor client routing directives precisely because this opacity is a real problem for teams trying to debug production issues.

What you can actually control

You cannot get an llm reproducibility guarantee from the API layer. You can reduce variance enough for most practical purposes.

Pin the model version explicitly

# Always use dated snapshots when available
model = "gpt-4o-mini-2024-07-18"  # Not "gpt-4o-mini"

If your provider does not offer dated models, treat the model identifier as a moving target.

Use temperature=0 and low top_p

params = {
    "temperature": 0,
    "top_p": 0.001,  # Effectively greedy but avoids numerical edge cases
    "max_tokens": 500,
}

This eliminates sampler variance. It does not eliminate logit variance.

Request logprobs and monitor drift

response = client.chat.completions.create(
    model="gpt-4o-mini-2024-07-18",
    messages=[...],
    temperature=0,
    logprobs=True,
    top_logprobs=5
)

# Track the top-1 logprob for each generated token
# Sudden drops indicate model or infrastructure changes
for token_logprob in response.choices[0].logprobs.content:
    print(token_logprob.token, token_logprob.logprob)

Logprob drift is your early warning system. If the same prompt produces the same tokens but with different confidence, the underlying model or hardware has changed.

Cache aggressively at the application layer

import hashlib
import json
from functools import lru_cache

def cache_key(messages, params):
    payload = json.dumps({"messages": messages, "params": params}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]

@lru_cache(maxsize=10000)
def cached_completion(key, messages, params):
    return client.chat.completions.create(model=params["model"], messages=messages, **params)

# Usage
key = cache_key(messages, params)
response = cached_completion(key, messages, params)

This is the only true reproducibility guarantee you can build: identical inputs return identical cached outputs. It also cuts cost and latency.

Evaluate with semantic equivalence, not string equality

from difflib import SequenceMatcher

def semantic_match(expected, actual, threshold=0.95):
    """Rough proxy. Replace with embedding similarity for production."""
    return SequenceMatcher(None, expected.strip(), actual.strip()).ratio() >= threshold

# In your eval harness
assert semantic_match(golden_output, actual_output), f"Drift detected: {actual_output}"

String equality tests will flake. Design your evaluations around functional correctness: does the code run, does the JSON parse, does the answer contain the required facts.

The decisive takeaway

An llm reproducibility guarantee does not exist and will not exist at the API layer. The stack from hardware kernels through batching, quantization, speculative decoding, and provider routing introduces irreducible variance. Seeds only control the final sampler. Model identifiers are not immutable contracts.

Build your systems assuming output will drift. Cache at the application layer for exact repeatability. Monitor logprobs for silent changes. Evaluate on semantic criteria, not string matching. Treat the LLM as a stochastic component with a loose SLA, not a deterministic function.

If you need true bitwise reproducibility, you must self-host on fixed hardware with deterministic algorithms enabled and accept the throughput penalty. For everyone else: design for variance, not against it.