If you ship code that calls multiple LLM providers, you need a way to test llm code without rate limits blowing up your CI pipeline. Provider sandboxes and free tiers throttle aggressively, and live calls make tests flaky, slow, and expensive. This guide walks through building a deterministic mock layer that speaks the OpenAI chat completions protocol so your client code stays unchanged across providers.
Step 1: Pin the API surface you actually depend on
Most multi-provider LLM clients converge on the OpenAI chat completions schema, even when they swap model names or add provider-specific headers. Before writing any test harness, list the exact requests your code makes: endpoint path, auth header format, body fields like model, messages, temperature, and the response fields you parse (choices[0].message.content, usage.total_tokens).
If you route through a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint for 240+ models and handles automatic fallback when a provider is degraded, your contract is just that single schema. You do not need to mock Anthropic or Mistral directly; you mock the gateway.
A typical client call looks like this:
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="test")
resp = client.chat.completions.create(
model="claude-3-opus",
messages=[{"role": "user", "content": "Summarize: ..."}],
temperature=0.2,
)
print(resp.choices[0].message.content)Your tests must exercise that exact create call without leaving the process.
Step 2: Stand up a local mock server
Use a minimal FastAPI app that implements POST /v1/chat/completions. It should return valid JSON in the OpenAI shape, and optionally emulate rate limits via a query parameter or header. This keeps your test llm code without rate limits interference because the mock never throttles—unless you ask it to.
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import JSONResponse
app = FastAPI()
@app.post("/v1/chat/completions")
async def chat(request: Request):
body = await request.json()
model = body.get("model", "gpt-4o")
if request.headers.get("x-simulate-429") == "true":
raise HTTPException(status_code=429, detail="Rate limited")
content = f"echo:{model}:{body['messages'][-1]['content']}"
return JSONResponse({
"id": "chatcmpl-mock",
"object": "chat.completion",
"model": model,
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": content},
"finish_reason": "stop"
}],
"usage": {"prompt_tokens": 5, "completion_tokens": 5, "total_tokens": 10}
})Run it with uvicorn mock_llm:app --port 8765. The mock returns a predictable string containing the model name, so your assertions can confirm routing.
Step 3: Redirect client calls to the mock in tests
Set the base URL through an environment variable that your client reads at construction. The OpenAI SDK respects OPENAI_BASE_URL. For multi-provider setups that wrap several SDKs, centralize the URL in one config module.
import os
from openai import OpenAI
def get_client():
base = os.getenv("LLM_TEST_BASE_URL", "https://api.openai.com/v1")
return OpenAI(base_url=base, api_key=os.getenv("LLM_API_KEY", "sk-test"))In your pytest conftest.py, start the mock server as a fixture (or use a separate process) and point the env var at it:
import pytest
import subprocess
import time
import requests
@pytest.fixture(scope="session")
def mock_server():
proc = subprocess.Popen(["uvicorn", "mock_llm:app", "--port", "8765"])
time.sleep(1.5)
while True:
try:
requests.get("http://localhost:8765/")
break
except Exception:
time.sleep(0.2)
yield "http://localhost:8765"
proc.terminate()Then in a test:
def test_claude_routing(mock_server, monkeypatch):
monkeypatch.setenv("LLM_TEST_BASE_URL", mock_server)
client = get_client()
resp = client.chat.completions.create(
model="claude-3-opus",
messages=[{"role": "user", "content": "hello"}],
)
assert "claude-3-opus" in resp.choices[0].message.contentThis is the core pattern to test llm code without rate limits: your code runs exactly as in production, but DNS never leaves localhost.
Step 4: Simulate multi-provider fallback
Real systems degrade. If your client implements fallback—or relies on a gateway that does—you need to assert that behavior. Two approaches:
Client-side fallback
Your code catches RateLimitError and retries with a different model. Mock the first call with a 429, then succeed.
def test_client_fallback(mock_server, monkeypatch):
monkeypatch.setenv("LLM_TEST_BASE_URL", mock_server)
client = get_client()
try:
client.chat.completions.create(
model="claude-3-opus",
messages=[{"role": "user", "content": "x"}],
extra_headers={"x-simulate-429": "true"}
)
except Exception as e:
assert e.status_code == 429
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "x"}],
)
assert "gpt-4o" in resp.choices[0].message.contentGateway-directed fallback
If you send routing hints, test that the mock honors them. For example, n4n.ai honors client routing directives and forwards provider cache-control hints. You can assert that your code sends x-routing: failover and the mock returns a different model’s content when the primary is mocked as degraded.
def test_gateway_failover_header(mock_server, monkeypatch):
monkeypatch.setenv("LLM_TEST_BASE_URL", mock_server)
client = get_client()
resp = client.chat.completions.create(
model="claude-3-opus",
messages=[{"role": "user", "content": "x"}],
extra_headers={"x-routing": "failover", "x-simulate-429": "true"}
)
assert resp.usage.total_tokens == 10Adjust the mock to inspect x-routing and return a non-429 response if failover is requested, emulating gateway behavior.
Step 5: Isolate CI from the real network
A mock is only safe if tests cannot accidentally hit production. Use the pytest-socket plugin to disable sockets except localhost, or set firewall rules in your CI container.
# pytest.ini
[pytest]
addopts = --disable-socketThen allow localhost explicitly in a fixture:
@pytest.fixture(autouse=True)
def allow_local(monkeypatch):
import socket
orig = socket.socket
def guarded(*args, **kwargs):
s = orig(*args, **kwargs)
orig_connect = s.connect
def connect(addr):
if addr[0] not in ("127.0.0.1", "::1"):
raise RuntimeError("Network blocked in tests")
return orig_connect(addr)
s.connect = connect
return s
monkeypatch.setattr(socket, "socket", guarded)Now any stray call to api.openai.com fails fast, surfacing bugs where you forgot to inject the mock URL.
Step 6: Verify success
Success means three things: (1) every test passes without external requests, (2) model routing and fallback logic are covered, (3) mocked usage fields propagate to your metering code.
Write a final integration test that runs a full multi-turn conversation across two providers and asserts token counts:
def test_multi_provider_conversation(mock_server, monkeypatch):
monkeypatch.setenv("LLM_TEST_BASE_URL", mock_server)
client = get_client()
for model in ["claude-3-opus", "gpt-4o-mini"]:
r = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "ping"}],
)
assert r.usage.total_tokens == 10
assert model in r.choices[0].message.contentRun pytest -q and confirm zero network calls and zero 429s from providers. If you previously fought flaky CI due to rate limits, you have now removed the only variable: the network. That is how you test llm code without rate limits in a multi-provider world.
A note on streaming
If your client uses stream=True, extend the mock to yield SSE chunks. The same routing and fallback tests apply; just parse text/event-stream in your client and assert the concatenated content matches the model tag. Keep the mock’s chunk size fixed to avoid timing flakes.
Cache-control headers
Some gateways forward provider cache-control hints. If your code sets extra_headers={"cache-control": "max-age=3600"}, assert the mock receives it. This ensures your caching strategy survives provider swaps.
The pattern holds for any number of providers: one local contract, deterministic responses, forced errors on demand. Your CI stays green when Anthropic throttles or OpenAI is down for maintenance—because your tests never knew they existed.