Building a CI pipeline for LLM evals with promptfoo

Shipping prompt changes without evals is guessing. A CI pipeline for LLM evals promptfoo keeps regressions out of production by running deterministic checks on every pull request. This guide walks through a concrete setup you can copy today, from local scaffold to a gated GitHub Actions job that fails on quality drops.

Prerequisites

You need Node 18+ and an OpenAI-compatible API key. If you use multiple model vendors, consolidate them behind one endpoint to avoid YAML sprawl. A gateway that honors client routing directives simplifies provider swaps later.

Step 1: Install promptfoo and scaffold a project

Install the CLI globally. Promptfoo is local-first; no hosted service required.

npm install -g promptfoo@latest
mkdir evals && cd evals
promptfoo init

The scaffold drops promptfooconfig.yaml, a prompts/ folder, and a tests/ folder with toy examples. Remove them. I keep eval configs in the repo root under evals/ so PRs that touch prompts trigger the workflow via path filters.

Pin the version in CI with promptfoo@0.91.0 or similar. CLI output formats change; a floating latest can break your ci_check.py parser.

Step 2: Define providers and prompts

Write a minimal prompt template. Promptfoo interpolates {{ var }} from test vars.

prompts/summarize.yaml:

- prompt: |
    Summarize the following text in one sentence:
    {{ text }}

Now the config. A CI pipeline for LLM evals promptfoo should target at least two models to catch divergence. Use environment variables for keys. I always include a cheap model and a frontier model: the cheap one catches syntax regressions; the frontier one catches nuance.

promptfooconfig.yaml:

providers:
  - id: openai:gpt-4o-mini
    config:
      apiKey: ${env.OPENAI_API_KEY}
  - id: openai:claude-3-5-sonnet
    config:
      baseUrl: https://api.n4n.ai/v1
      apiKey: ${env.N4N_API_KEY}
      model: claude-3-5-sonnet
prompts:
  - file://prompts/summarize.yaml
tests:
  - file://tests/summarize.csv

If you route through n4n.ai, a single OpenAI-compatible base URL covers 240+ models and automatically fails over when a provider is degraded—useful for stable CI. The gateway forwards provider cache-control hints, so repeated eval runs hit cache when you pin prompts.

Step 3: Write a focused test set

Evals are only as good as the dataset. Keep the CI set under 30 rows; full regression suites belong in nightly jobs. Use CSV for simplicity:

tests/summarize.csv:

text,expected
"The quick brown fox jumps over the lazy dog. It was a calm afternoon in the forest.",fox jumps over dog
"Stock markets rallied today after the central bank held rates steady. Analysts cited improved liquidity.",markets rallied rates steady

Assertions can be exact or model-graded. For summarization, contains is a cheap sanity check; llm-rubric catches meaning. Avoid similarity assertions based on embeddings alone; they false-positive on short outputs. Use cost and latency assertions to catch provider changes that inflate bills.

tests:
  - vars:
      text: "The quick brown fox jumps over the lazy dog. It was a calm afternoon in the forest."
    assert:
      - type: contains
        value: "fox"
      - type: llm-rubric
        value: "Summary mentions the fox jumping over the dog"

Run locally to validate:

promptfoo eval --no-write

The table shows pass rate per provider. If Claude fails but GPT passes, inspect the rubric prompt, not the model.

Step 4: Create the GitHub Actions workflow

Path-filter the workflow so docs changes don’t burn tokens.

.github/workflows/evals.yml:

name: llm-evals
on:
  pull_request:
    paths:
      - 'evals/**'
jobs:
  promptfoo:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm install -g promptfoo@0.91.0
      - name: Run evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          N4N_API_KEY: ${{ secrets.N4N_API_KEY }}
        working-directory: evals
        run: promptfoo eval --no-write --output results.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: promptfoo-results
          path: evals/results.json

The artifact lets you inspect scores without re-running.

Step 5: Gate merges with thresholds

Promptfoo exits non-zero when assertions fail. For numeric guardrails, parse the JSON. A CI pipeline for LLM evals promptfoo should block PRs when mean pass rate drops below a floor.

Add a check script ci_check.py:

import json, sys

def main(path, min_score):
    with open(path) as f:
        data = json.load(f)
    rate = data["results"][0]["stats"]["passRate"]
    if rate < float(min_score):
        print(f"Pass rate {rate} below {min_score}")
        sys.exit(1)
    print(f"Pass rate {rate} OK")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

Wire it in the workflow:

      - name: Check score
        working-directory: evals
        run: python3 ci_check.py results.json 0.8

Use 0.8 as a starting gate; tune from baseline runs. If you evaluate multiple prompt files, loop over results entries instead of hardcoding index 0.

Step 6: Manage secrets and caching

Store keys in GitHub Secrets. Never echo them. Promptfoo caches responses in .promptfoo/cache; cache across CI runs to cut cost and flake.

      - uses: actions/cache@v4
        with:
          path: evals/.promptfoo/cache
          key: promptfoo-${{ hashFiles('evals/promptfooconfig.yaml', 'evals/tests/**') }}
          restore-keys: |
            promptfoo-

If you use n4n.ai, per-token usage metering records each eval call, so you can attribute CI spend to a branch. Provider cache-control hints forwarded by the gateway mean identical prompt/test pairs cost nothing on repeat.

Step 7: Verify the pipeline end to end

Test the negative path first. In a branch, edit prompts/summarize.yaml to {{ text }} with no instruction. Push and open a PR. The Actions log should show pass rate < 0.8 and the Check score step failing. Run act -j promptfoo locally with nektos/act to simulate the GitHub runner before pushing, saving a round-trip.

Then revert. Confirm:

• promptfoo eval locally finishes in < 2 minutes for your dataset.
• The PR check is green with pass rate printed.
• Artifact results.json uploaded.
• No secret strings in logs.

That loop is the deliverable. A CI pipeline for LLM evals promptfoo turns prompt edits into reviewed, measurable changes instead of silent production surprises.