Postman collections for multi-provider LLM API testing

Testing against multiple LLM providers from a single workspace saves time and exposes provider drift early. A well-structured postman collection multi provider llm setup lets you swap endpoints, model names, and auth tokens without rewriting requests, and gives you a repeatable harness for regression checks.

Use environments to isolate provider config

Don’t hardcode URLs or keys in requests. Create one Postman environment per provider or per gateway, and store base_url, api_key, and model as variables. This keeps the requests themselves provider-agnostic and makes the collection portable across teammates.

{
  "name": "OpenAI-compat",
  "values": [
    { "key": "base_url", "value": "https://api.openai.com/v1", "enabled": true },
    { "key": "api_key", "value": "{{vault:openai_key}}", "enabled": true },
    { "key": "model", "value": "gpt-4o-mini", "enabled": true }
  ]
}

Switching from OpenAI to a local vLLM instance or an Anthropic-compatible proxy is then a one-click environment change. The same request template works unchanged. Use Postman’s variable scopes: collection variables for logical names, environment variables for secrets and hosts. Never commit the environment file with real keys to git; use the Postman vault or a local untracked file.

Build one canonical chat completion request

The core of any postman collection multi provider llm workflow is a single OpenAI-compatible request. Most providers now mirror /chat/completions, so you can standardize on that shape and avoid per-vendor request bodies.

POST {{base_url}}/chat/completions
Content-Type: application/json
Authorization: Bearer {{api_key}}

{
  "model": "{{model}}",
  "messages": [{"role": "user", "content": "Return a JSON object with key 'ok'."}],
  "temperature": 0.1
}

If you use a gateway that fronts many models, the model string may include a provider prefix (e.g., anthropic/claude-3-5-sonnet). Keep that mapping in the environment, not in the request body. Note that some providers require extra headers like anthropic-version when called directly; an OpenAI-compatible translation layer removes that burden. Decide whether your collection tests the raw provider or the compatibility layer—mixing both in one folder creates confusion.

Parameterize routing and cache hints

When a gateway honors client routing directives, you can force a specific provider with a header. This is useful to verify that each backend returns compatible shapes without maintaining separate folders.

// Pre-request script on the request
const force = pm.environment.get("force_provider") || "openai";
pm.request.headers.upsert({ key: "x-router-force", value: force });
// Forward provider cache-control if the gateway supports it
pm.request.headers.upsert({ key: "x-cache-control", value: "ttl=300" });

Set force_provider per environment or via a collection variable when running a matrix. This approach collapses a tree of provider-specific requests into one parameterized call. If the gateway forwards provider cache-control hints, you can assert that repeated calls within the TTL return x-cache-hit: true to validate caching.

Assert on response shape and usage

Postman tests run in a sandboxed JavaScript runtime. Write tight assertions that catch provider-specific deviations early—fields renamed, usage omitted, or content truncated.

pm.test("status 200", () => pm.response.to.have.status(200));
const body = pm.response.json();
pm.test("message content is string", () => {
  pm.expect(body.choices[0].message.content).to.be.a("string");
});
pm.test("usage reported", () => {
  pm.expect(body.usage.total_tokens).to.be.a("number");
});
pm.environment.set("last_completion", body.choices[0].message.content);

Capture last_completion to feed a follow-up turn. This builds a multi-turn conversation test without leaving Postman. For stricter contracts, load a JSON schema and validate body with ajv inside the test script; this catches silent schema drift when a provider upgrades its API.

Chain requests for multi-turn and tool calls

Create a second request that reads the previous answer. Run both as a folder with sequential execution.

{
  "model": "{{model}}",
  "messages": [
    {"role": "user", "content": "Return a JSON object with key 'ok'."},
    {"role": "assistant", "content": "{{last_completion}}"},
    {"role": "user", "content": "Now validate that object and say yes/no."}
  ]
}

If you need tool calls, assert that tool_calls appears and that your mock endpoint echoes the right arguments. Keep the mock inside the collection using a {{mock_url}} variable pointing at Postman’s built-in mock server. This isolates the test from external dependencies and makes the suite deterministic.

Exercise fallback and degradation

Providers fail in production. Your test harness should simulate that. If you point the collection at a single OpenAI-compatible endpoint that aggregates providers—such as n4n.ai, which offers automatic fallback when a provider is rate-limited and per-token metering—you can force a degraded path by setting a header that routes to a provider you’ve temporarily disabled upstream, or by watching x-routed-provider response headers.

pm.test("fallback header present", () => {
  pm.expect(pm.response.headers.get("x-routed-provider")).to.be.a("string");
});

Without a gateway, you must script each provider’s error response manually: send a bad key to provider A, expect 401, then repeat with provider B. That’s brittle; prefer a routing layer that surfaces which backend actually served the token. You can also use Postman’s mock server to return a 429 with Retry-After and confirm your client logic backs off.

Run the suite in CI with Newman

Postman collections are portable. Export the collection and environment, then run with Newman in your pipeline.

newman run llm-tests.json -e openai-compat.json --timeout-request 60000

Use --env-var to override force_provider for a matrix job. This catches provider regressions before deploy. Keep the collection in repo, but reference secrets from CI variables, not the environment file.

Common pitfalls

Secret leakage. Never paste API keys into the collection JSON. Use Postman’s secret variables or environment files excluded from git.

Model name drift. gpt-4o on one provider is not the same as claude-3-5-sonnet. Maintain a lookup table in a pre-request script that maps a logical name to the provider-specific string.

Timeouts. Default Postman timeout is 10s. LLM completions often exceed that. Set request timeout to 60s in settings, or per-request via the UI.

Streaming. If stream: true, the response is SSE, not JSON. Either disable streaming in tests or parse text/event-stream in the test script. Most regression checks don’t need streaming.

Header case. Some gateways are case-sensitive on routing headers. Standardize on lowercase and use upsert to avoid duplicates.

Schema laxness. Don’t assert only on status 200. Providers can return 200 with empty choices. Assert on content length and usage.

Tradeoffs: Postman vs code

A postman collection multi provider llm artifact is excellent for sharing with non-engineers and for exploratory debugging. Export it and run with Newman in CI for smoke tests. But for load testing, token-cost analysis, or complex branching, a Python pytest harness with httpx is stronger. Insomnia offers similar environment/variable features if your team prefers its UI, but the collection format is less ubiquitous.

When you export your postman collection multi provider llm setup, you get a JSON file that documents your integration surface. Treat it like code: review changes, version it, and keep environments separate from the collection. That discipline turns a casual testing tool into a reliable multi-provider contract suite.