Automating LLM API tests with Postman's collection runner

Most teams validate their LLM integrations with handwritten scripts that rot the moment the schema changes. The postman collection runner llm api approach gives you a version-controlled, shareable contract test that exercises chat completions, auth, and error handling from one place. Below is the exact workflow we use to keep our gateway clients honest.

Step 1: Set up the Postman environment and variables

Start by isolating configuration from the requests. Create a Postman environment with three variables: baseUrl, apiKey, and model. Keeping these external means the same collection runs against local mocks, staging, or production without edits.

Export the environment as JSON so you can commit it (minus secrets) and load it in CI:

{
  "name": "LLM Test Env",
  "values": [
    { "key": "baseUrl", "value": "https://api.openai.com/v1", "enabled": true },
    { "key": "apiKey", "value": "REPLACE_IN_CI", "enabled": true },
    { "key": "model", "value": "gpt-4o-mini", "enabled": true }
  ]
}

In practice you never hard-code the key. In CI you overwrite apiKey with a secret; locally you use Postman’s variable scoping or the Vault integration. The baseUrl should point at any OpenAI-compatible endpoint. That compatibility matters because it lets the same tests target different backends.

Step 2: Author the core chat completion request

Create a request named ChatCompletion. Set method to POST and URL to {{baseUrl}}/chat/completions. Add the auth header:

Authorization: Bearer {{apiKey}}
Content-Type: application/json

Use a minimal body that forces deterministic output. Low temperature and a tiny max_tokens keep test latency and cost down:

{
  "model": "{{model}}",
  "messages": [
    { "role": "user", "content": "Reply with the single word: pong" }
  ],
  "max_tokens": 5,
  "temperature": 0
}

This is the contract you care about: given a well-formed request, the endpoint returns a choices array and a usage block. If the provider changes the schema, this request breaks loudly.

Step 3: Write response assertions in the test tab

Postman runs the JavaScript in the Tests tab after each response. Assert the shape you depend on, not just status 200. For an LLM gateway, the critical fields are choices[0].message.content and usage.total_tokens.

pm.test("status is 200", () => pm.response.to.have.status(200));

const body = pm.response.json();

pm.test("has at least one choice", () => {
  pm.expect(body.choices).to.be.an('array').with.length.greaterThan(0);
});

pm.test("choice has message content", () => {
  pm.expect(body.choices[0].message.content).to.be.a('string').with.length.greaterThan(0);
});

pm.test("usage metering present", () => {
  pm.expect(body.usage).to.have.property('total_tokens');
  pm.expect(body.usage.total_tokens).to.be.a('number').greaterThan(0);
});

pm.test("latency under 5s", () => {
  pm.expect(pm.response.responseTime).to.be.below(5000);
});

The latency assertion catches regressions in provider routing. The usage assertion confirms the endpoint honors per-token metering—a requirement if you bill downstream customers.

Step 4: Parameterize across models and providers

A single model test is weak. The postman collection runner llm api pattern shines when you feed it a data file that iterates over models. Create models.json:

[
  { "model": "gpt-4o-mini" },
  { "model": "claude-3-haiku-20240307" },
  { "model": "mistral-small-latest" }
]

In the Collection Runner, select this file under Data. Postman binds each model property to {{model}} per iteration. If you point baseUrl at an OpenAI-compatible gateway such as n4n.ai, the same collection hits 240+ models and picks up automatic fallback when a provider is rate-limited, turning your postman collection runner llm api check into a multi-provider canary.

One caveat: some providers reject unknown parameters. Keep the body to the common subset (model, messages, max_tokens, temperature) unless you maintain per-provider collections.

Step 5: Test error paths and rate limits

Happy-path tests lie by omission. Duplicate the request, name it ChatCompletion_InvalidKey, and override the auth header to Bearer bad-key. Assert the failure mode:

pm.test("rejects bad key with 401", () => {
  pm.response.to.have.status(401);
});

const err = pm.response.json();
pm.test("error has message", () => {
  pm.expect(err).to.have.property('error');
  pm.expect(err.error).to.have.property('message');
});

Add a third request with a malformed body (e.g., messages missing) to confirm a 400. These tests verify your client surfaces provider errors correctly instead of crashing on parse.

Step 6: Execute with the Collection Runner UI and Newman

Inside Postman, open Runner, pick the collection and environment, attach models.json, and run. You’ll see per-iteration pass/fail and response times.

For automation, use Newman (Postman’s CLI). Install and run:

npm install -g newman

newman run llm-api-tests.postman_collection.json \
  -e llm-test-env.postman_environment.json \
  -d models.json \
  --reporters cli,json \
  --reporter-json-export newman-out.json

Newman exits non-zero if any assertion fails, which is what CI needs. The JSON report captures responseTime and failed assertions for trend tracking.

Step 7: Embed in CI (GitHub Actions)

Commit the collection, environment template, and data file to tests/. Store the real key as a repo secret. A minimal workflow:

name: llm-api-contract-tests
on: [push, pull_request]

jobs:
  postman:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      - run: npm install -g newman
      - name: Run LLM API collection
        run: |
          newman run tests/llm-collection.json \
            -e tests/env.json \
            -d tests/models.json \
            --env-var "apiKey=${{ secrets.LLM_API_KEY }}" \
            --env-var "baseUrl=${{ secrets.LLM_BASE_URL }}"
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          LLM_BASE_URL: ${{ secrets.LLM_BASE_URL }}

The --env-var flags inject secrets at runtime without writing them to disk. If you use a gateway that honors client routing directives, you can also pass a header variable to force a specific provider and confirm fallback logic separately.

Step 8: Verify success and interpret reports

Success means Newman prints 0 failures and exits with code 0. In the JSON report, inspect run.stats.assertions for any partial failures and run.timings.responseAvg for latency drift.

Concrete verification checklist:

• All iterations report status is 200 on the happy path.
• usage.total_tokens is a positive integer on every model in the data file.
• The invalid-key request returns 401 and the malformed-body request returns 400.
• Average response time stays under your SLO (we use 5s for small prompts).

If those hold, your postman collection runner llm api suite is enforcing the contract. When a provider silently drops a field or bumps a version, the run goes red before your users notice.

Advanced: streaming and cache-control

Non-streaming tests cover 90% of integration bugs. For streaming, Postman’s native SSE support is limited; we shell out to curl in a separate CI step to assert text/event-stream chunks and usage in the final event. If your gateway forwards provider cache-control hints (some OpenAI-compatible gateways do), add an assertion on the x-cache response header to confirm caching works as designed. That is a separate test, not a replacement for the collection above.

Treat the collection as code. Review changes in PRs, rotate keys via secrets, and bump models.json when you add a provider. That discipline is what keeps LLM integrations from quietly breaking in production.