Setting up GitHub Actions to test prompts on every PR

Most LLM app regressions slip through because prompts change without any automated check. Setting up GitHub Actions prompt testing on pull requests catches those changes with executable assertions before they hit production.

Why prompt testing belongs in CI

Prompt edits are code edits. They alter model behavior, often silently. A typo in a system message can drop JSON mode compliance; a reordered few-shot example can change extraction accuracy. Treating prompts as static assets that bypass review is how teams lose trust in their own pipelines.

GitHub Actions prompt testing on pull requests forces every prompt modification through the same scrutiny as a function change. You get a diff, a run, and a green check. The alternative is finding out that customer support summaries suddenly omit refund amounts during a Friday deploy.

Step 1: Structure your prompts and tests

Keep prompts as versioned files, not inline strings scattered across services. A flat layout works for small repos and scales reasonably:

prompts/
  extract_entities.v1.json
  summarize_ticket.v2.json
tests/
  test_extract_entities.py
  test_summarize_ticket.py

Each prompt file holds the template and metadata. Use a templating convention you control:

{
  "model": "gpt-4o-mini",
  "temperature": 0,
  "messages": [
    {"role": "system", "content": "Extract entities as JSON with key 'entities'."},
    {"role": "user", "content": "Text: {{input}}"}
  ]
}

Write a pytest test that loads the file, renders the template, calls the model, and asserts on the shape. Isolate the client creation so you can swap endpoints in one place:

import json, os, openai

client = openai.OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL")  # defaults to OpenAI if unset
)

def _render(spec, text):
    return [{"role": m["role"], "content": m["content"].replace("{{input}}", text)}
            for m in spec["messages"]]

def test_extract_entities_returns_json():
    with open("prompts/extract_entities.v1.json") as f:
        spec = json.load(f)
    resp = client.chat.completions.create(
        model=spec["model"],
        temperature=spec["temperature"],
        messages=_render(spec, "Apple bought Siri in 2010")
    )
    content = resp.choices[0].message.content
    data = json.loads(content)  # raises if not valid JSON
    assert "entities" in data

This is a minimal smoke test. Expand with domain assertions and negative cases (empty input, adversarial input).

Step 2: Use an endpoint that survives provider outages

Live LLM calls in CI fail for reasons unrelated to your code: provider 429s, regional degradation, key quota. If your prompt tests go red because OpenAI is rate-limited, engineers start ignoring the check.

Point your test client at an OpenAI-compatible gateway that handles fallback. n4n.ai exposes one endpoint covering 240+ models and automatically reroutes when a provider is degraded, so a transient upstream error doesn’t fail your build. Per-token metering also keeps CI spend visible instead of a mystery line on a bill.

Configure the base URL and key through environment variables:

client = openai.OpenAI(
    api_key=os.environ["N4N_API_KEY"],
    base_url="https://api.n4n.ai/v1"
)

The rest of the code stays identical because the interface is OpenAI-compatible. You can pin a specific model in the prompt spec or let the gateway apply its default routing.

Step 3: Write the GitHub Actions workflow

Create .github/workflows/prompt-tests.yml. Trigger on pull requests to main, and only run when prompt or test files change to save minutes.

name: prompt-tests
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'tests/**'
      - '.github/workflows/prompt-tests.yml'
concurrency:
  group: prompt-tests-${{ github.ref }}
  cancel-in-progress: true
jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install openai pytest pytest-json-report tenacity
      - name: Run prompt tests
        env:
          N4N_API_KEY: ${{ secrets.N4N_API_KEY }}
          LLM_BASE_URL: https://api.n4n.ai/v1
        run: pytest tests/ --json-report --json-report-file=report.json -q
      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: prompt-test-report
          path: report.json

Store the gateway key in repository secrets. Never hardcode it. The concurrency block cancels superseded runs on the same branch, which matters when you push fixup commits rapidly.

Step 4: Make assertions robust to non-determinism

Temperature 0 reduces variance but doesn’t eliminate it. Add structural checks rather than exact string matches. For example, verify that extracted entities include known spans:

def test_extract_entities_contains_known():
    with open("prompts/extract_entities.v1.json") as f:
        spec = json.load(f)
    resp = client.chat.completions.create(
        model=spec["model"], temperature=spec["temperature"],
        messages=_render(spec, "Apple bought Siri in 2010"))
    data = json.loads(resp.choices[0].message.content)
    labels = [e["name"] for e in data["entities"]]
    assert "Apple" in labels
    assert "Siri" in labels

If you need stricter evaluation, call a second model as a grader with a fixed rubric. Keep the grader call in a separate test so a grader hiccup doesn’t block the smoke test. Absorb intermittent parsing errors with a retry:

import tenacity

@tenacity.retry(stop=tenacity.stop_after_attempt(3), wait=tenacity.wait_fixed(2))
def call_model(spec, text):
    return client.chat.completions.create(
        model=spec["model"], temperature=spec["temperature"],
        messages=_render(spec, text))

Step 5: Report results inline

A red check is useful; a red check with the actual model output is actionable. Write a step summary directly in the workflow run UI:

      - name: Summarize
        if: always()
        run: |
          echo "### Prompt test summary" >> $GITHUB_STEP_SUMMARY
          jq -r '.tests[] | "- \(.nodeid): \(.outcome)"' report.json >> $GITHUB_STEP_SUMMARY

For deeper visibility, comment on the PR using the GitHub CLI:

      - name: Comment on PR
        if: failure()
        env:
          GH_TOKEN: ${{ github.token }}
          PR: ${{ github.event.pull_request.number }}
        run: |
          jq -r '.tests[] | select(.outcome=="failed") | "- \(.nodeid)"' report.json > fails.txt
          gh pr comment $PR --body "Prompt tests failed:$(cat fails.txt)"

Reviewers see which prompt failed and why without cloning the repo.

Step 6: Enforce and verify

Go to Settings → Branches in GitHub, add a branch protection rule for main, and require the prompt-tests workflow to pass. This closes the loop: no prompt change merges without a run.

To verify success, make a deliberate breaking edit to a prompt file on a branch—for example, change the system message to forbid JSON output—and open a PR. The workflow should trigger, run the tests, and fail with a clear summary. Then revert the edit; the re-run goes green. That cycle confirms the gate works.

Step 7: Scale the suite without slowing PRs

As prompts multiply, keep PR latency under a few minutes. Split the suite: fast smoke tests on every PR, full evaluation on a nightly cron or manual trigger. Use actions/cache for pip dependencies:

      - uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}

Use small models for smoke tests and reserve large models for nightly batches. Set explicit timeouts on every job. If you call the gateway, leverage provider cache-control hints by marking static prompt prefixes as cached; the gateway forwards those hints, cutting both latency and token cost on repeated runs.

GitHub Actions prompt testing on pull requests is not a luxury once you ship LLM features weekly. It is the cheapest line of defense against silent behavior drift.