Debugging inconsistent outputs from identical prompts

You sent the same prompt twice and got different JSON shapes, different facts, or different tone. Inconsistent llm outputs same prompt is rarely random luck; it’s a signal that something in your call stack—sampling params, provider routing, or hidden nondeterminism—is leaking variance into production. This guide gives you an ordered, code-backed path to find the leak and seal it.

Step 1: Capture the exact request bytes

Most “identical” prompts aren’t byte-identical. A missed header, a floating temperature, or a reordered message list will change the completion. Before blaming the model, prove the request is constant.

Serialize the full request—URL, headers, body, model string, and every sampling parameter—then hash it. If the hash moves, your client is the bug.

import hashlib, json, openai

def request_fingerprint(req: dict, headers: dict) -> str:
    payload = {
        "body": req,
        "headers": {k.lower(): v for k, v in headers.items()},
    }
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

client = openai.OpenAI(base_url="https://api.openai.com/v1")
req_body = {
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Extract name and age: John, 42"}],
    "temperature": 0.7,
}
headers = {"authorization": f"Bearer {client.api_key}"}
print(request_fingerprint(req_body, headers))

Wire this into a small logging middleware so every call emits its fingerprint to your log store. In a distributed system, attach the hash to your trace context (OpenTelemetry) so you can correlate diverging outputs with the exact bytes sent.

What to watch for

• messages order: some SDKs sort, some don’t.
• response_format toggles that silently change parsing.
• user field or session IDs that providers may use for routing.
• Default parameters injected by older SDK versions (e.g., max_tokens=16 if unset).

Verify success: Two calls you believe are identical produce the same SHA-256. If not, fix the caller before touching the model.

Step 2: Kill sampling variance

Temperature and top_p are the usual suspects. Set temperature=0 and top_p=1 to force greedy decoding. If the API supports a seed parameter, pin it.

req_body = {
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": PROMPT}],
    "temperature": 0,
    "top_p": 1,
    "seed": 12345,
}

Note: seed improves repeatability but is not a hard guarantee across model versions or providers. Some endpoints ignore it; others apply it only to the first token. Also zero out frequency_penalty and presence_penalty. If your client library exposes top_k, set it to a large number or disable it.

When you see inconsistent llm outputs same prompt even at temperature zero, the cause is almost always outside the sampler: model weight swaps, batch scheduling, or fallback routing.

Don’t trust “deterministic” marketing

A vendor claiming “deterministic mode” still runs on shared hardware with concurrent batches. Logits can be computed in different order across CUDA graphs. Treat zero-temperature as necessary but not sufficient.

Verify success: Execute the same request 10 times in a loop. With temperature=0 and a respected seed, outputs should be character-for-character identical. If they still diverge, sampling isn’t your only problem.

Step 3: Audit the prompt for hidden drift

Dynamic system messages, injected timestamps, or whitespace normalization can silently alter completions. Canonicalize and assert.

def canonical_prompt(messages):
    out = []
    for m in messages:
        # collapse all whitespace, strip
        content = " ".join(m["content"].split())
        out.append(f'{m["role"]}:{content}')
    return "|".join(out)

assert canonical_prompt(call_1.messages) == canonical_prompt(call_2.messages)

Common leaks:

• datetime.now() in system prompts (“Today is …”).
• Retrieval augmentations that reorder contexts between calls.
• Unicode vs ASCII hyphens, or BOM characters in file-loaded prompts.
• Template engines that insert random request IDs for tracing.

Chat template mismatches

If you self-host or use open-weight models, the chat template applied by the client library can differ between versions. Pin the library version and dump the rendered token stream.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
print(tok.apply_chat_template(messages, tokenize=False))

Verify success: The assertion passes, and a hex dump of the tokenized prompt matches across runs.

Step 4: Detect provider or model-version swaps

A model name like claude-3-sonnet or gpt-4-turbo is a label, not a checksum. Providers rotate weights, snapshots, or region-specific builds. Your “same prompt” might hit a different tensor on Tuesday.

If you sit behind a gateway, routing logic can change mid-session. For example, inconsistent llm outputs same prompt often appears when a fallback kicks in. If you route through n4n.ai, automatic fallback when a provider is rate-limited or degraded can silently switch the backing model even if your requested name stays constant. Honor client routing directives (x-n4n-route) or pin a specific provider to avoid that.

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "x-n4n-route: openai:gpt-4o-mini" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

Also log the system_fingerprint field (OpenAI) or equivalent provider response header. Some providers expose a build hash; capture it. If you’re using open-weight models behind your own proxy, record the git commit of the weights repo.

Multi-region surprises

Even with one provider, us-east-1 and eu-west-1 can run different hotfixes. Pin the region if the API allows.

Verify success: The fingerprint or provider tag is identical across the two calls that previously diverged. If the tag changed, you’ve found the leak.

Step 5: Inspect decoding boundaries

Truncation at max_tokens or an early stop sequence produces fragments that look like different answers. Check finish_reason.

resp = client.chat.completions.create(**req_body)
fr = resp.choices[0].finish_reason
print(fr)  # "stop" | "length" | "content_filter"

If you see "length", you’re comparing a partial completion to a full one. Raise max_tokens or remove conflicting stop strings. A content_filter finish means the provider muted the output—also nondeterministic relative to your expectation.

Streaming artifacts

When streaming, the final token can be split differently across calls due to network chunking. Disable streaming for the debugging harness.

Verify success: All test calls return finish_reason == "stop" with no mid-string cuts.

Step 6: Run a repetition harness

Automate the check. A minimal harness calls the endpoint N times and reports exact-match rate and semantic distance.

import collections, openai

client = openai.OpenAI()
def sample():
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"user","content":"List 3 capitals"}],
        temperature=0, seed=1)
    return r.choices[0].message.content.strip()

outputs = collections.Counter(sample() for _ in range(20))
print(outputs.most_common())

If you get 20 identical strings, you’ve fixed it. If you see a spread, the cause is upstream of your code (provider nondeterminism or versioning). For structured output, parse each response and compare the dict, not the raw string—whitespace differences are not real divergence.

Measuring semantic drift

If exact match fails but answers are equivalent, use a grounded metric:

from difflib import SequenceMatcher
sim = SequenceMatcher(None, out_a, out_b).ratio()

Set a threshold (e.g., 0.98) for “acceptable” consistency.

Verify success: Exact-match rate is 100% on 20 runs, or the divergence is explained by a known provider limitation documented in their status page.

Step 7: Lock consistency into the pipeline

For production, treat the model as a deterministic function only when you constrain it:

• Pin temperature=0, top_p=1, and a seed if supported.
• Pin the model snapshot via provider version suffix or routing header.
• Forward provider cache-control hints to avoid recomputation drift; gateways that honor cache-control and client routing make this explicit.
• Log the request hash and response fingerprint on every call.

req_body = {
    "model": "gpt-4o-mini-2024-07-18",  # snapshot pin
    "messages": MSG,
    "temperature": 0,
    "seed": 42,
    "extra_headers": {"cache-control": "max-age=3600"}
}

Add a synthetic canary: every hour, send a fixed prompt and assert the output matches the stored golden response within a tolerance. If you use a gateway with per-token usage metering, you can also alert on sudden token-count variance for the same prompt as a proxy for decoding changes.

Verify success: Over a 24-hour canary with 1000 repeated prompts, the exact-match rate stays above your SLO (e.g., 99.9%). Alert if it drops.

Inconsistent llm outputs same prompt is debuggable. The fix is usually discipline in the request layer, not a different model.