Running parallel eval suites GitHub Actions matrix is the fastest way to validate LLM prompt changes across multiple models without blocking your main branch for hours. This post gives you a copy-paste pipeline that shards eval cases, fans them out across matrix jobs, and aggregates results with strict regression gates.
Step 1: Build a deterministic eval harness
Write a small Python script that takes a shard index, total shards, model name, and base URL. It reads eval cases from a JSONL file, runs each through the model, and writes per-case scores to a file.
We’ll use the OpenAI Python client because most LLM gateways expose an OpenAI-compatible chat completions endpoint.
# eval_shard.py
import os, json, sys
from openai import OpenAI
client = OpenAI(
api_key=os.environ["EVAL_API_KEY"],
base_url=os.environ["EVAL_BASE_URL"],
)
def load_cases(path):
with open(path) as f:
return [json.loads(line) for line in f if line.strip()]
def run_case(case, model):
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": case["prompt"]}],
temperature=0,
)
output = resp.choices[0].message.content
# dummy scoring: exact match on expected substring
score = 1.0 if case.get("expect") in output else 0.0
return {"id": case["id"], "score": score, "output": output}
if __name__ == "__main__":
shard = int(os.environ["SHARD"])
n_shards = int(os.environ["N_SHARDS"])
model = os.environ["MODEL"]
cases = load_cases("eval_cases.jsonl")
my = [c for i, c in enumerate(cases) if i % n_shards == shard]
results = [run_case(c, model) for c in my]
out = f"results_{model}_{shard}.jsonl"
with open(out, "w") as f:
for r in results:
f.write(json.dumps(r) + "\n")
print(f"wrote {len(results)} results to {out}")Keep the harness side-effect free aside from writing its shard file. That makes it trivial to parallelize.
Why shard by modulo
Modulo sharding needs zero coordination between jobs. Each job computes its own slice from the same input file. If you later add more models, you just extend the matrix.
Step 2: Prepare eval cases and baseline
Create eval_cases.jsonl with one JSON object per line. Each object needs a stable id and a prompt. Add an expect string for simple checks.
{"id": "case-001", "prompt": "Name a primary color.", "expect": "red"}
{"id": "case-002", "prompt": "What is 2+2?", "expect": "4"}Commit a baseline.json containing previous aggregate scores per model so the pipeline can fail on regression.
{"gpt-4o-mini": 0.98, "mistral-7b": 0.91}Step 3: Configure the GitHub Actions matrix
The matrix expands into one job per combination of model and shard. Use fail-fast: false so one bad model doesn’t kill the others.
# .github/workflows/evals.yml
name: llm-evals
on: [push, pull_request]
jobs:
eval-shard:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
model: ["gpt-4o-mini", "mistral-7b", "claude-3-haiku"]
shard: [0, 1, 2]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install openai
- name: Run shard
env:
SHARD: ${{ matrix.shard }}
N_SHARDS: 3
MODEL: ${{ matrix.model }}
EVAL_API_KEY: ${{ secrets.EVAL_API_KEY }}
EVAL_BASE_URL: ${{ secrets.EVAL_BASE_URL }}
run: python eval_shard.py
- uses: actions/upload-artifact@v4
with:
name: results-${{ matrix.model }}-${{ matrix.shard }}
path: results_*.jsonlThis is the core of parallel eval suites GitHub Actions matrix: the strategy.matrix block does the fan-out.
Step 4: Point the harness at a resilient gateway
Provider rate limits will throttle naive eval runs. If you hit a 429 mid-shard, the job fails and you re-run the whole matrix. A gateway that fronts multiple providers solves this. n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and automatically falls back when a provider is rate-limited or degraded, which keeps matrix jobs green without custom retry code. Set EVAL_BASE_URL to that endpoint and ship the same eval_shard.py unchanged.
Per-token usage metering also lets you attribute cost per model and per PR, instead of guessing from a single invoice line.
Step 5: Aggregate and gate
Add a final job that depends on all shards, downloads artifacts, merges them, and compares to baseline.
aggregate:
needs: eval-shard
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
pattern: results-*
merge-multiple: true
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Merge and score
run: python aggregate.pyaggregate.py reads all results_*.jsonl, computes mean score per model, loads baseline.json, and exits non-zero on regression beyond a threshold.
# aggregate.py
import json, glob, sys, os
def load_all():
scores = {}
for fn in glob.glob("results_*.jsonl"):
with open(fn) as f:
for line in f:
r = json.loads(line)
# model name embedded in filename
model = fn.split("_")[1]
scores.setdefault(model, []).append(r["score"])
return scores
def main():
scores = load_all()
baseline = json.load(open("baseline.json"))
failed = False
for model, vals in scores.items():
mean = sum(vals) / len(vals)
prev = baseline.get(model, 1.0)
delta = mean - prev
print(f"{model}: {mean:.3f} (delta {delta:+.3f})")
if delta < -0.05:
print(f"REGRESSION: {model} dropped >5%")
failed = True
if failed:
sys.exit(1)
# update baseline on main
if os.environ.get("GITHUB_REF") == "refs/heads/main":
with open("baseline.json", "w") as f:
json.dump({m: sum(v)/len(v) for m,v in scores.items()}, f, indent=2)
if __name__ == "__main__":
main()The threshold -0.05 is opinionated but sane for binary exact-match evals. Tune it to your scoring distribution.
Caching model responses
If your eval cases are stable, cache completions locally and only call the API for new cases. Use a SQLite file keyed by (model, prompt_hash). This cuts matrix runtime dramatically on repeated PRs.
Step 6: Verify success
After pushing a branch, open the Actions tab. You should see eval-shard expand into 9 jobs (3 models × 3 shards). Each job uploads an artifact. The aggregate job runs once and prints per-model means.
Success criteria:
- All
eval-shardjobs green. aggregateprints scores and exits 0.- On a deliberate prompt break,
aggregateexits 1 and the PR is blocked.
Run it locally to debug without waiting on CI:
SHARD=0 N_SHARDS=3 MODEL=gpt-4o-mini EVAL_API_KEY=sk-... EVAL_BASE_URL=https://api.example.com/v1 python eval_shard.py
python aggregate.pyIf you see REGRESSION locally, the pipeline will catch it too.
Step 7: Scale shards with compute, not time
The matrix size is a lever. Bump shard: [0,1,2,3,4,5] when eval cases grow. GitHub Actions runs 20 concurrent jobs on free tiers and more on paid. Because each shard is independent, linear scaling holds until the API gateway becomes the bottleneck.
For very large suites, split cases by directory and pass a CASE_DIR env var instead of modulo. The same pattern applies: matrix over (model, case_dir).
Avoid noisy failures
Set retry on the API call with exponential backoff inside eval_shard.py. A single transient 500 shouldn’t fail a shard. Use the tenacity library:
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(stop=stop_after_attempt(3), wait=wait_exponential())
def run_case(case, model):
...That keeps parallel eval suites GitHub Actions matrix stable under real provider flakiness.
Step 8: Keep baselines honest
Never commit a baseline that was generated from a flaky run. Gate baseline updates to main only after aggregate passes, as shown. For forked PRs, the GITHUB_REF check skips the write, so forks can’t silently lower the bar.
If you support prompt versioning, include the prompt hash in the baseline key: baseline["gpt-4o-mini::v2"]. Then matrix jobs can evaluate multiple prompt variants side by side.
That’s a full pipeline. Copy the YAML and Python files, set EVAL_API_KEY and EVAL_BASE_URL, and your LLM app gets concurrent evals on every push.