How to test LLM outputs when results aren't reproducible

Testing non-reproducible LLM outputs requires a fundamentally different mindset than traditional software testing. You cannot assert exact string equality when the system under test samples from a probability distribution. Instead, you need deterministic sampling controls, statistical evaluation harnesses, property-based assertions that capture structural correctness, and continuous monitoring to catch drift before users do. This guide walks through each layer, from local development to production observability.

Step 1: Lock down what you can control

Start by eliminating variance sources that are not the model itself. Temperature, top-p, and top-k are the obvious knobs, but the seed parameter is the one that actually buys you reproducibility — when the provider honors it.

# openai_client.py
from openai import OpenAI
import os

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def complete_deterministic(
    prompt: str,
    model: str = "gpt-4o-mini",
    seed: int = 42,
    temperature: float = 0.0,
    max_tokens: int = 512,
) -> str:
    """
    Request deterministic completion. Not all providers honor seed;
    treat this as best-effort and verify in Step 2.
    """
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
        top_p=1.0,
        seed=seed,
        max_tokens=max_tokens,
    )
    return resp.choices[0].message.content

Set temperature=0.0 and a fixed seed. Most major providers (OpenAI, Anthropic, Google) now respect seed for a given model version, but the contract is “best effort.” Model updates, infrastructure changes, or provider-side batching can still produce different tokens. Treat deterministic sampling as a variance reduction technique, not a guarantee.

Verify success: Run the same prompt 10 times with the same seed. If you get identical outputs, the provider honored the seed for that model version. Log the model fingerprint (system_fingerprint in OpenAI responses) to detect silent model upgrades.

Step 2: Build a statistical evaluation harness

When exact matching fails, you need to evaluate distributions. Define a metric that captures task correctness, then measure its mean and confidence interval over N samples.

# eval_harness.py
import statistics
from dataclasses import dataclass
from typing import Callable
from openai_client import complete_deterministic

@dataclass
class EvalResult:
    metric_name: str
    mean: float
    stdev: float
    ci95_low: float
    ci95_high: float
    n: int
    passed: bool

def evaluate_distribution(
    prompt: str,
    metric_fn: Callable[[str], float],
    n_samples: int = 30,
    threshold: float = 0.8,
    model: str = "gpt-4o-mini",
    seed_base: int = 42,
) -> EvalResult:
    """
    Sample N times with different seeds, compute metric per sample,
    return mean + 95% CI. Passes if CI lower bound >= threshold.
    """
    scores = []
    for i in range(n_samples):
        # Vary seed per sample to explore the distribution
        output = complete_deterministic(prompt, model=model, seed=seed_base + i)
        scores.append(metric_fn(output))

    mean = statistics.mean(scores)
    stdev = statistics.stdev(scores) if n_samples > 1 else 0.0
    # 95% CI via t-distribution approximation
    margin = 1.96 * stdev / (n_samples ** 0.5)
    ci_low, ci_high = mean - margin, mean + margin

    return EvalResult(
        metric_name=metric_fn.__name__,
        mean=mean,
        stdev=stdev,
        ci95_low=ci_low,
        ci95_high=ci_high,
        n=n_samples,
        passed=ci_low >= threshold,
    )

Choose metrics that reflect your actual task:

• Exact match / F1 / ROUGE for extraction and summarization
• Code execution pass rate for code generation
• Schema validation pass rate for structured output
• Custom rubric scored by a stronger model (LLM-as-judge)

Verify success: Run the harness on a known-good prompt. The CI lower bound should clear your threshold with comfortable margin. If the CI straddles the threshold, increase n_samples or improve the prompt.

Step 3: Add property-based tests for structural guarantees

Statistical metrics miss structural failures — invalid JSON, missing required fields, hallucinated enum values. Property-based testing (à la Hypothesis) generates diverse inputs and asserts invariants that must hold for every output.

# test_properties.py
import json
import pytest
from hypothesis import given, strategies as st, settings
from openai_client import complete_deterministic
from pydantic import BaseModel, ValidationError

class ExtractionSchema(BaseModel):
    entities: list[str]
    relations: list[tuple[str, str, str]]
    confidence: float

SCHEMA_PROMPT = """
Extract entities and relations from the text. Return ONLY valid JSON matching:
{"entities": ["..."], "relations": [["subj", "pred", "obj"]], "confidence": 0.0-1.0}
Text: {text}
"""

@settings(max_examples=50, deadline=None)
@given(text=st.text(min_size=10, max_size=500))
def test_schema_validity(text: str):
    """Every output must parse as valid JSON and validate against schema."""
    prompt = SCHEMA_PROMPT.format(text=text)
    output = complete_deterministic(prompt, seed=123, temperature=0.0)
    
    # Property 1: Valid JSON
    parsed = json.loads(output)
    
    # Property 2: Schema compliance
    validated = ExtractionSchema(**parsed)
    
    # Property 3: Confidence in range
    assert 0.0 <= validated.confidence <= 1.0
    
    # Property 4: Relations reference declared entities
    entity_set = set(validated.entities)
    for subj, _, obj in validated.relations:
        assert subj in entity_set, f"Subject {subj} not in entities"
        assert obj in entity_set, f"Object {obj} not in entities"

@settings(max_examples=20, deadline=None)
@given(text=st.text(min_size=10, max_size=500))
def test_idempotence_at_temp_zero(text: str):
    """Same prompt + same seed + temp=0 should produce identical output."""
    prompt = SCHEMA_PROMPT.format(text=text)
    out1 = complete_deterministic(prompt, seed=42, temperature=0.0)
    out2 = complete_deterministic(prompt, seed=42, temperature=0.0)
    assert out1 == out2, "Deterministic sampling not honored"

Run these in CI with a small sample budget. They catch regressions that statistical metrics miss: a prompt change that breaks JSON syntax, a model upgrade that drops a required field, a provider that stops honoring seed.

Verify success: CI passes consistently. Introduce a deliberate bug (remove a required field from the prompt) and confirm the test fails.

Step 4: Create golden datasets with fuzzy matching

Golden datasets are your regression anchor. Store (input, expected_output) pairs, but compare with fuzzy matching — not exact strings. Use semantic similarity or task-specific equivalence.

# golden_dataset.py
import json
from pathlib import Path
from dataclasses import dataclass
from typing import Callable
from openai_client import complete_deterministic
from sentence_transformers import SentenceTransformer

EMBEDDER = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

@dataclass
class GoldenCase:
    name: str
    input: str
    expected: str
    threshold: float = 0.85  # cosine similarity threshold

def load_golden_dataset(path: Path) -> list[GoldenCase]:
    with open(path) as f:
        data = json.load(f)
    return [GoldenCase(**item) for item in data]

def cosine_similarity(a: str, b: str) -> float:
    emb_a = EMBEDDER.encode(a, normalize_embeddings=True)
    emb_b = EMBEDDER.encode(b, normalize_embeddings=True)
    return float((emb_a * emb_b).sum())

def run_golden_evaluation(
    cases: list[GoldenCase],
    model: str = "gpt-4o-mini",
    seed: int = 42,
) -> dict:
    results = {}
    for case in cases:
        output = complete_deterministic(case.input, model=model, seed=seed)
        score = cosine_similarity(output, case.expected)
        results[case.name] = {
            "passed": score >= case.threshold,
            "score": score,
            "threshold": case.threshold,
            "output": output,
        }
    return results

Store golden cases as JSON:

[
  {
    "name": "extract_entities_simple",
    "input": "Extract entities: Apple acquired Beats for $3B in 2014.",
    "expected": "Entities: Apple, Beats. Relation: Apple acquired Beats.",
    "threshold": 0.85
  },
  {
    "name": "summarize_technical",
    "input": "Summarize in 2 sentences: The transformer architecture uses self-attention...",
    "expected": "Transformers use self-attention to process sequences in parallel. This enables training on massive datasets.",
    "threshold": 0.80
  }
]

Semantic similarity via embeddings handles paraphrase variance. For code or structured output, replace cosine_similarity with a task-specific comparator (AST equivalence, JSON deep-diff with ignored fields, SQL execution equivalence).

Verify success: All golden cases pass. When you change a prompt, run the golden set first — any regression shows up immediately.

Step 5: Implement LLM-as-judge for nuanced quality

Some qualities (tone, helpfulness, safety) resist programmatic metrics. Use a stronger model as a calibrated judge. The key is structured output and few-shot calibration.

# llm_judge.py
from pydantic import BaseModel, Field
from openai_client import complete_deterministic
import json

class JudgeScore(BaseModel):
    score: int = Field(ge=1, le=5)
    reasoning: str

JUDGE_PROMPT = """You are an expert evaluator. Score the response on {criterion} from 1-5.
Return ONLY JSON: {{"score": int, "reasoning": "..."}}

Criterion: {criterion}
Definition: {definition}

Examples:
{examples}

Response to evaluate:
{response}
"""

def judge_response(
    response: str,
    criterion: str,
    definition: str,
    examples: list[tuple[str, int, str]],
    model: str = "gpt-4o",
    seed: int = 42,
) -> JudgeScore:
    ex_text = "\n".join(
        f'Response: "{r}"\nScore: {s}\nReasoning: "{rea}"'
        for r, s, rea in examples
    )
    prompt = JUDGE_PROMPT.format(
        criterion=criterion,
        definition=definition,
        examples=ex_text,
        response=response,
    )
    output = complete_deterministic(prompt, model=model, seed=seed, temperature=0.0)
    return JudgeScore.model_validate_json(output)

Calibrate with 5-10 examples spanning the score range. Run the judge over your golden set outputs to establish baseline scores. Track judge scores over time alongside your programmatic metrics.

Verify success: Judge scores on golden set are stable (variance < 0.5 points across runs). Human spot-check agrees with judge ≥ 90% of the time.

Step 6: Monitor production drift with shadow evaluation

Testing non-reproducible LLM outputs doesn’t stop at deploy. You need continuous visibility into whether the model’s behavior has shifted. Shadow evaluation runs your evaluation pipeline on a sample of production traffic.

# shadow_eval.py
import random
import time
from dataclasses import dataclass
from typing import Optional
from openai_client import complete_deterministic
from eval_harness import evaluate_distribution

@dataclass
class ShadowConfig:
    sample_rate: float = 0.01  # 1% of requests
    min_samples_per_hour: int = 10
    alert_threshold_ci_drop: float = 0.05  # CI lower bound drop

class ShadowEvaluator:
    def __init__(self, config: ShadowConfig, metric_fn, threshold: float):
        self.config = config
        self.metric_fn = metric_fn
        self.threshold = threshold
        self.baseline_ci_low: Optional[float] = None

    def maybe_evaluate(self, prompt: str, production_output: str) -> Optional[dict]:
        if random.random() > self.config.sample_rate:
            return None

        # Re-run with fixed seed to get distribution
        result = evaluate_distribution(
            prompt=prompt,
            metric_fn=self.metric_fn,
            n_samples=self.config.min_samples_per_hour,
        )

        if self.baseline_ci_low is None:
            self.baseline_ci_low = result.ci95_low
            return {"status": "baseline_established", "ci_low": result.ci95_low}

        drop = self.baseline_ci_low - result.ci95_low
        alert = drop > self.config.alert_threshold_ci_drop

        return {
            "status": "evaluated",
            "current_ci_low": result.ci95_low,
            "baseline_ci_low": self.baseline_ci_low,
            "drop": drop,
            "alert": alert,
        }

Wire this into your request path (async, non-blocking). Alert when the CI lower bound drops more than 5 percentage points from baseline. This catches silent model upgrades, provider degradation, or prompt injection attacks that degrade quality without breaking the API.

Verify success: Deploy shadow evaluation for one week. Baseline stabilizes. Inject a known-bad prompt variant and confirm alert fires.

Step 7: Automate the full loop in CI/CD

Wire everything into a pipeline that runs on every prompt or model change:

# .github/workflows/llm-eval.yml
name: LLM Evaluation
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'eval/**'
  schedule:
    - cron: '0 6 * * *'  # daily drift check

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -r requirements.txt
      - name: Run property tests
        run: pytest test_properties.py -v
      - name: Run golden dataset
        run: python -m eval_golden
      - name: Run statistical harness
        run: python -m eval_statistical
      - name: Run LLM judge
        run: python -m eval_judge
      - name: Comment PR with results
        uses: actions/github-script@v7
        with:
          script: |
            // post summary to PR

Require all checks to pass before merge. The daily scheduled run catches provider-side changes (model updates, infrastructure shifts) that no code change triggered.

Verify success: Open a PR that degrades a prompt. CI fails. Check the daily run logs — they should show stable baselines with no alerts.

Putting it together: a mental model for testing non-reproducible LLM outputs

Start with deterministic sampling and property tests — they’re fast, cheap, and catch the loudest failures. Add golden datasets once you have stable prompts. Layer statistical harnesses and LLM judges for the quality dimensions that matter to your product. Shadow evaluation is the safety net that catches everything else.

The n4n.ai gateway forwards provider system_fingerprint and cache-control headers, which makes it easier to correlate evaluation results with specific model versions and detect when a provider silently swaps the underlying model. If your gateway doesn’t surface these, you’re flying blind on reproducibility.

Testing non-reproducible LLM outputs is not about eliminating variance — it’s about measuring it, bounding it, and alerting when it crosses your tolerance. Build the harness once, run it everywhere, and treat every model change (yours or the provider’s) as a potential regression.