What is promptfoo? A guide to prompt regression testing

What is promptfoo? It’s an open-source command-line and library tool for evaluating LLM prompts, models, and application flows through declarative test cases and assertions. In short, promptfoo brings unit-test discipline to prompt engineering, letting you catch regressions before they hit production.

How promptfoo works

Promptfoo separates three concerns: the prompts under test, the providers that execute them, and the test cases that assert expected behavior. You declare these in a promptfooconfig.yaml file (or via TS/JS config) and run promptfoo eval.

Configuration anatomy

A minimal config has four top-level keys: prompts, providers, tests, and optionally defaultTest. Prompts can be inline strings, file references, or templated functions. Providers are identifiers like openai:gpt-4o or anthropic:messages:claude-3-5-sonnet. Tests are arrays of input/assert pairs.

prompts:
  - "Summarize the following text in one sentence: {{text}}"
providers:
  - openai:gpt-4o-mini
tests:
  - vars:
      text: "Long article about climate change..."
    assert:
      - type: contains
        value: "climate"

Variables use Handlebars syntax. You can reference {{var}} inside prompts or pass them as JSON. This keeps your test inputs separate from prompt templates, so you can reuse one prompt across hundreds of cases.

Running evaluations

The CLI compiles each prompt/provider/test combination into a single inference call. It executes them, collects outputs, and evaluates assertions. Results render as a markdown table in the terminal and as an HTML report with promptfoo view.

npx promptfoo eval -c promptfooconfig.yaml

For CI, use promptfoo eval --quiet --output report.json and fail the build on assertion errors. The exit code is non-zero when any assertion fails, which maps cleanly to GitHub Actions or GitLab CI.

Assertions and metrics

Assertions are the core of regression testing. Built-in types include equals, contains, regex, similar (embedding cosine), llm-rubric (a model grades the output), and javascript for custom logic. You can set thresholds on numeric metrics like cost and latency.

assert:
  - type: llm-rubric
    value: "Output must not mention the competitor brand"
  - type: latency
    threshold: 2000

The javascript assertion receives output, vars, and context and must return a boolean or throw. This is where you encode domain rules that a simple string match can’t capture.

Why promptfoo matters for regression testing

LLM outputs are non-deterministic. A prompt tweak that improves one query can silently degrade another. Without a test suite, you rely on manual spot checks that don’t scale.

Non-determinism and prompt drift

Even with temperature 0, provider API changes, model version shifts, and retrieval corpus updates alter outputs. Promptfoo locks a set of representative inputs and asserts invariants. When a change breaks an assertion, you see it immediately.

Cost and latency regressions

A new prompt might raise token usage by 30% or push latency past your SLA. Promptfoo records per-call cost and duration. You can assert cost < 0.01 or latency < 1500 to catch silent bloat.

CI integration

Treat prompt configs as code. Store promptfooconfig.yaml in the repo, run eval on pull requests, and block merges that drop pass rate. This is the same muscle memory as unit tests, applied to natural language interfaces.

A concrete example

Suppose you maintain a support ticket classifier. The prompt maps a ticket to a department. You want to ensure “refund” goes to Billing and “password” to Account.

Sample config

prompts:
  - file://classifier.prompt
providers:
  - openai:gpt-4o-mini
tests:
  - vars:
      ticket: "I need a refund for order #123"
    assert:
      - type: equals
        value: "Billing"
  - vars:
      ticket: "I forgot my password"
    assert:
      - type: equals
        value: "Account"
  - vars:
      ticket: "Your product is awesome"
    assert:
      - type: javascript
        value: "output !== 'Billing' && output !== 'Account'"

The classifier.prompt file contains:

Classify the support ticket into one of: Billing, Account, General.
Ticket: {{ticket}}
Department:

Running and interpreting

After promptfoo eval, the table shows each test’s pass/fail. If a model update starts returning “Billing Department” instead of “Billing”, the equals assertion fails. You then either tighten the prompt to constrain output format or relax the assertion to contains.

This loop is exactly what what is promptfoo answers for teams shipping LLM features: a safety net that turns “looks good to me” into a red/green build.

Common misconceptions about promptfoo

It’s just a prompt debugger

Promptfoo is not a playground like the OpenAI console. It’s a test runner. You can use it to debug, but its value is in repeated, automated checks across model or prompt versions.

It replaces model evaluation

Model evaluation (perplexity, MMLU, human preference) is a different layer. Promptfoo tests your specific prompts against your specific cases. It does not rank foundation models globally; it tells you whether your prompt still works on the model you pinned.

It’s only for OpenAI

The openai: provider is common, but promptfoo supports Anthropic, Google, local models via Ollama, and any HTTP endpoint that speaks OpenAI-compatible chat format. If you route through a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint for 240+ models with automatic fallback when a provider is degraded, you can point promptfoo’s openai provider at that base URL and rotate models by changing a single config value.

It’s heavyweight

A useful suite can be ten test cases in a 30-line YAML file. You don’t need a vector store or a database. Start small, add cases as you find edge failures.

It gives false confidence

A test suite only covers the cases you wrote. Promptfoo won’t catch a hallucination on an input you didn’t enumerate. Treat the suite as a regression guard, not a correctness proof.

Using promptfoo with multiple model providers

Because promptfoo treats providers as strings, A/B testing across models is trivial. List multiple providers and the same tests run against each.

providers:
  - openai:gpt-4o-mini
  - anthropic:messages:claude-3-5-haiku
  - openai:gpt-4o

The report compares cost, latency, and assertion pass rates side by side. This is how you answer “should we switch models?” with data instead of vibes.

Anatomy of a good regression suite

Start with happy-path cases that must always work. Add adversarial inputs that previously broke production. Include a few llm-rubric checks for tone or safety. Keep the config in version control and review changes to tests as seriously as code changes.

A mature suite might have 50+ tests across 3 providers, run on every commit. The pass rate becomes a product metric.

Getting started checklist

1. Install: npm install -g promptfoo or use npx promptfoo.
2. Create promptfooconfig.yaml with one prompt, one provider, three tests.
3. Run promptfoo eval and read the table.
4. Add llm-rubric assertions for fuzzy requirements.
5. Wire promptfoo eval into your CI workflow.

What is promptfoo if not the cheapest insurance against silent LLM regressions? It’s a disciplined practice encoded in a config file, and it belongs in every LLM project’s toolchain.