Prompt tweaks and model swaps are code changes with behavioral consequences. Treating them like any other diff means you need a continuous evaluation LLM evals pull request pipeline that runs the same battery of tests on every proposed change. Below is a concrete setup that catches regressions before they merge, using open-source tooling and an OpenAI-compatible inference layer.
Step 1: Pin your eval corpus in git
Start with a small, high-signal dataset. Store it as JSONL so it diffs cleanly and survives code review:
{"id": "sql_1", "input": "List users who signed up in 2023", "expect": {"type": "sql", "contains": ["SELECT", "WHERE", "created_at"]}}
{"id": "sum_1", "input": "Summarize: The quick brown fox jumps over the lazy dog.", "expect": {"max_words": 10}}
{"id": "json_1", "input": "Return a JSON object with keys name, age", "expect": {"parse_json": true, "keys": ["name", "age"]}}Keep the corpus under evals/corpus.jsonl. When product requirements shift, you update the corpus in the same PR that changes prompts. This makes the eval set part of the review, not an external artifact owned by a separate team.
Write a loader that your test harness can reuse:
import json, pathlib
def load_cases(path="evals/corpus.jsonl"):
lines = pathlib.Path(path).read_text().splitlines()
return [json.loads(l) for l in lines if l.strip()]Corpus design rules
- Prefer executable assertions (
contains,max_words,parse_json) over fuzzy similarity. Fuzzy checks lie. - One case should target exactly one failure mode. A case that mixes SQL validity and tone is two cases.
- Never put secrets or PII in the corpus. Use synthetic substitutes.
Verify success: python -c "import evals; print(len(evals.load_cases()))" prints the case count. If it grows or shrinks in a PR, reviewers see exactly why.
Step 2: Write a deterministic harness
Use pytest and an OpenAI-compatible client. Set temperature=0 and a fixed seed where the model supports it. The harness calls the model and checks expectations.
import os, pytest, openai
client = openai.OpenAI(
base_url=os.environ["OPENAI_BASE_URL"],
api_key=os.environ["OPENAI_API_KEY"],
)
def run_model(prompt, model):
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0,
seed=42,
)
return resp.choices[0].message.content
@pytest.mark.parametrize("case", load_cases(), ids=lambda c: c["id"])
def test_case(case):
model = os.environ["EVAL_MODEL"]
out = run_model(case["input"], model)
exp = case["expect"]
if "contains" in exp:
for token in exp["contains"]:
assert token.lower() in out.lower(), f"missing {token}"
if "max_words" in exp:
assert len(out.split()) <= exp["max_words"], "too long"
if exp.get("parse_json"):
obj = json.loads(out) # raises if invalid
for k in exp.get("keys", []):
assert k in obj, f"missing key {k}"This is a real test. It fails loudly when a prompt change breaks SQL generation or inflates summaries. Do not reach for an LLM-as-judge in the critical path until you have 20 deterministic cases that already block merges.
Step 3: Establish a baseline on main
Before gating PRs, know your current numbers. Run the suite against main and dump results:
OPENAI_BASE_URL=https://api.openai.com/v1 EVAL_MODEL=gpt-4o \
pytest evals/ -q --json-report --json-report-file=baseline.jsonCommit baseline.json or store it as a CI artifact on the main branch. Your continuous evaluation LLM evals pull request job will compare against this artifact so it knows what “good” looked like before the diff.
Branching strategy
Run baseline generation nightly on main rather than per-commit to avoid drift from transient provider issues. If the nightly baseline regresses, that is a separate alert—do not let it block PRs until you have triaged it.
Step 4: Add the CI workflow
GitHub Actions is enough. Create .github/workflows/evals.yml:
name: llm-evals
on:
pull_request:
paths:
- "evals/**"
- "src/prompts/**"
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: "3.11"}
- run: pip install openai pytest pytest-json-report
- name: Download baseline
uses: actions/download-artifact@v4
with:
name: baseline
path: ./
continue-on-error: true
- run: |
OPENAI_BASE_URL=${{ secrets.OPENAI_BASE_URL }} \
OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY }} \
EVAL_MODEL=${{ vars.EVAL_MODEL }} \
pytest evals/ -q --json-report --json-report-file=pr.json
- name: Compare to baseline
run: python evals/compare.py baseline.json pr.jsonThe compare.py script fails the build if any case that passed on baseline now fails, or if aggregate score drops beyond a threshold you define (e.g., more than 2% absolute drop).
Step 5: Route through a resilient gateway
CI runners are ephemeral and often share IPs, so provider rate limits bite exactly when you batch evals. Instead of hardcoding api.openai.com, point OPENAI_BASE_URL at a gateway that aggregates providers. For example, n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models and automatically falls back when a provider is rate-limited or degraded, so your eval run does not stall on a 429 from a single vendor.
client = openai.OpenAI(
base_url="https://api.n4n.ai/v1", # single endpoint, many models
api_key=os.environ["N4N_KEY"],
)
# client routing directives forwarded; cache-control hints honoredYou keep the same chat.completions.create call. The gateway handles failover and per-token metering, which keeps eval cost visible per run. If you want to force a specific provider for reproducibility, pass the routing header the gateway documents; otherwise let it pick the cheapest healthy route.
Step 6: Gate merges on diffs
The comparison script should compute a regression delta. A minimal version:
import json, sys
def outcomes(path):
data = json.load(open(path))
return {t["nodeid"]: t["outcome"] == "passed" for t in data["tests"]}
base = outcomes("baseline.json")
pr = outcomes("pr.json")
regressions = [k for k in base if base[k] and not pr.get(k)]
if regressions:
print("REGRESSIONS:", regressions)
sys.exit(1)
base_pass = sum(base.values()) / len(base)
pr_pass = sum(pr.values()) / len(pr)
if base_pass - pr_pass > 0.02:
print(f"Pass rate dropped {base_pass:.2%} -> {pr_pass:.2%}")
sys.exit(1)Wire this as a required check in branch protection. Now a PR that breaks three SQL cases cannot be merged until the corpus is updated or the prompt fixed. This is the core of continuous evaluation LLM evals pull request discipline: every change is measured against the same bar.
Surface results in the PR
Use the GitHub API or a simple echo step to post a comment with per-case status. Engineers should not have to open the Actions log to see which eval failed.
Step 7: Control flakiness and cost
Model outputs are non-deterministic even at temperature 0 across providers. Mitigate:
- Set
seedandtemperature=0in the request, but treat a single failure as suspicious, not fatal. - Retry idempotent checks twice with backoff before marking a case failed.
- Cache prompt completions in the gateway (forward
cache_controlheaders) to cut token spend on static system prompts. - Keep the corpus small but adversarial; 50 cases that break easily beat 5000 trivial ones.
If you use a gateway that honors cache-control hints, prefix static system prompts with a cache breakpoint so repeated eval runs hit provider caches. That turns a 500-case run from a $2 line item into a few cents.
Verify success
A working pipeline shows the following in a PR:
- The
llm-evalsjob runs and posts a comment with pass/fail per case. - Merging a prompt change that drops SQL correctness fails the compare step.
- Reverting the prompt makes the check green again.
Run it locally with the same env vars to reproduce. Once this loop is tight, you can extend the corpus without fear, because the continuous evaluation LLM evals pull request gate will catch the next regression before your users do. If a case proves consistently flaky, move it to a separate “experimental” suite that reports but does not block, then fix the harness.