Setting pass/fail thresholds for LLM regression tests

Defining pass fail thresholds for LLM regression tests is the difference between a CI check that catches prompt drift and one that floods you with false alarms. This guide walks through a concrete pipeline for setting those thresholds against real model outputs, with code you can run today.

Step 1: Snapshot a baseline with a pinned model

You cannot set meaningful thresholds without a reference point. Lock the model version, temperature, and top_p before generating your baseline corpus. Model providers quietly shift weights and system behavior; an unpinned model makes every threshold lie.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

def generate(prompt: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-2024-05-13",
        temperature=0,
        top_p=1.0,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content

prompts = ["Summarize: ...", "Classify: ...", "Extract: ..."]
with open("baseline.jsonl", "w") as f:
    for p in prompts:
        f.write(json.dumps({"prompt": p, "output": generate(p)}) + "\n")

Store the raw outputs, not scores. You will recompute distributions as your eval code matures. If you cache responses at the HTTP layer, mark the cache key with the exact model string so a later re-run does not silently mix snapshots.

Step 2: Choose evaluation metrics that match the task

Threshold setting is only as good as the metric feeding it. For structured extraction, JSON validity and required-field presence beat fuzzy similarity. For open-ended answers, use deterministic checks (substring, regex) first and reserve model-as-judge for cases where nothing else works.

import json, re

def score_json_validity(output: str) -> float:
    try:
        json.loads(output)
        return 1.0
    except ValueError:
        return 0.0

def score_substring(output: str, must_contain: list[str]) -> float:
    if not must_contain:
        return 1.0
    return sum(1 for s in must_contain if s in output) / len(must_contain)

def score_regex(output: str, pattern: str) -> float:
    return 1.0 if re.search(pattern, output) else 0.0

Keep metrics as pure functions with no network calls. A judge model belongs in a separate, cached step so threshold tuning stays a sub-second local operation. If you must use semantic similarity, pick a fixed embedding model and store the vectors; never call it inline during threshold sweeps.

Step 3: Assemble a golden set with human-verified references

Thresholds need a target distribution. Build a golden set of 50–200 examples where a human has confirmed expected behavior. Size depends on task variance; classification needs less, free-form generation needs more. Store references alongside the prompt and any deterministic constraints.

{"prompt": "Extract name from: Ada Lovelace wrote notes", "reference": {"name": "Ada"}, "must_contain": ["Ada"], "regex": "name\":\\s*\"Ada\""}

Run your current prompt against this set and record both baseline and candidate scores. Treat the golden set as code: review additions in PRs, and never let a flaky example stay in the set without a comment explaining why.

Step 4: Compute score distributions to set pass fail thresholds for LLM regression tests

Dump per-metric scores into a NumPy array and inspect percentiles. The pass fail thresholds for LLM regression tests should sit where the baseline distribution tails off, not at the mean. A threshold at the median will fail half your historically good outputs.

import numpy as np

baseline_scores = np.array([0.92, 1.0, 0.85, 0.99, 0.80, 1.0, 0.97])
p1, p5, p50, p95 = np.percentile(baseline_scores, [1, 5, 50, 95])
print(f"p1={p1} p5={p5} p50={p50} p95={p95}")

If the 5th percentile is 0.8, setting the pass bar at 0.8 means 5% of historically acceptable outputs would fail. Set the floor at p1 or p2 for the first cut, then tighten after the suite proves stable across multiple model revisions.

{
  "json_validity": {"min_pass": 0.98},
  "substring_match": {"min_pass": 0.9},
  "regex_match": {"min_pass": 0.95}
}

These numbers are configuration, not logic. Engineers adjust them in review, not by editing test assertions. When you promote a threshold change, attach the percentile output that justified it.

Step 5: Encode the threshold check in a CI harness

Write a pytest suite that loads the config, calls the model, and asserts against the floor. Fail the build when the aggregate score drops below threshold. Use a fixture to load the golden set and baseline together so the test compares like-for-like.

import json, pytest
from my_eval import score_json_validity, score_substring, score_regex

CONFIG = json.load(open("thresholds.json"))

@pytest.fixture
def regression_set():
    with open("golden.jsonl") as f:
        return [json.loads(line) for line in f]

def test_json_validity(regression_set):
    scores = [score_json_validity(o["output"]) for o in regression_set]
    assert sum(scores)/len(scores) >= CONFIG["json_validity"]["min_pass"]

def test_substring(regression_set):
    scores = [score_substring(o["output"], o["must_contain"]) for o in regression_set]
    assert sum(scores)/len(scores) >= CONFIG["substring_match"]["min_pass"]

Run it locally before pushing:

pytest tests/regression.py -q

If the suite is green on baseline and red on a deliberately broken prompt (e.g., truncate the system message), your thresholds have signal. Add a scheduled CI job that runs the suite nightly so silent drift surfaces even when no one opens a PR.

Step 6: Execute the suite across model revisions and providers

Prompts regress when the underlying model changes. Run the same harness against new model snapshots and alternate providers. Routing through n4n.ai gives one OpenAI-compatible endpoint that addresses 240+ models and automatically falls back when a provider is rate-limited, so a provider outage does not read as a prompt regression.

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ.get("LLM_BASE_URL", "https://gateway.n4n.ai/v1"),
    api_key=os.environ["LLM_API_KEY"],
)

def generate(prompt: str, model: str) -> str:
    resp = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content

Parameterize the model name in pytest with an env var or a small matrix. Compare score deltas per model; if gpt-4o-mini drops 10% on substring match but claude-3-haiku holds, your threshold caught a real routing risk, not a prompt bug. Keep per-model thresholds only if the task genuinely favors one provider—otherwise a single cross-model floor is easier to maintain.

Step 7: Treat thresholds as living configuration

Model behavior drifts. Review thresholds monthly with the eval set owner. If the golden set grows, recompute percentiles and raise the floor only when the baseline distribution supports it. A threshold tightened without new data is just a flake generator.

Add a changelog entry for every threshold edit. A diff like min_pass: 0.9 -> 0.85 is a QA decision, not a typo fix. Store the percentile report from Step 4 in the same PR so the reasoning survives staff turnover.

When you upgrade a model, run the regression suite against the new version before merging any prompt changes. The delta report should show whether the threshold held or needs a one-time reset because the new model shifted the baseline distribution upward.

Verifying success

A working setup passes on the pinned baseline, fails when you inject a known-bad prompt variant, and emits a per-metric score report in CI logs. Run the suite against the baseline weekly to confirm the thresholds still reflect reality. When a model upgrade ships, the regression run should show the delta before you merge the prompt change. If your pass fail thresholds for LLM regression tests survive a provider fallback event without a false red, the pipeline is doing its job.