Testing LLM API integrations in CI without real calls is essential for fast, deterministic, and cost-controlled pipelines. If you test llm api in ci without real calls by hitting live providers, you inherit rate limits, flaky responses, and unpredictable token charges. This guide shows how to build a mock layer that mirrors the OpenAI-compatible contract your code expects, then wire it into pytest and GitHub Actions.
Step 1: Pin the exact request and response contract
Most LLM clients speak the OpenAI chat completions shape. Your first job is to write down the minimal JSON your production code sends and receives. Do not mock a vague “AI response” — mock the real schema.
A typical request:
{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Summarize: ..."}],
"temperature": 0.2,
"stream": false
}A typical response:
{
"id": "chatcmpl-123",
"object": "chat.completion",
"model": "gpt-4o-mini",
"choices": [
{"index": 0, "message": {"role": "assistant", "content": "Summary: ..."}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}
}Encode this in code so both client and mock share a source of truth. A pydantic model works well:
from pydantic import BaseModel
class ChatRequest(BaseModel):
model: str
messages: list[dict]
temperature: float = 0.7
stream: bool = False
class ChatResponse(BaseModel):
id: str
object: str = "chat.completion"
model: str
choices: list[dict]
usage: dictIf you call a gateway that aggregates models, the same shape applies. For example, n4n.ai exposes one OpenAI-compatible endpoint across 240+ models, so your mock must accept any model string and echo it back. The contract is stable; only the routing behind it changes.
Step 2: Build a local mock server
Stand up a tiny HTTP server in your test process or as a sidecar. FastAPI is ideal because it validates types and runs async.
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, StreamingResponse
import asyncio
app = FastAPI()
async def stream_tokens():
for chunk in ['data: {"choices":[{"delta":{"content":"M"}}]}', 'data: {"choices":[{"delta":{"content":"OCK"}}]}', 'data: [DONE]']:
yield chunk + "\n\n"
await asyncio.sleep(0.01)
@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
body = await request.json()
if body.get("stream"):
return StreamingResponse(stream_tokens(), media_type="text/event-stream")
return JSONResponse({
"id": "mock-1",
"object": "chat.completion",
"model": body.get("model", "unknown"),
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": "MOCK_SUMMARY"},
"finish_reason": "stop"
}],
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}
})Run it on a fixed port in CI: uvicorn mock_server:app --port 8080. Your test code points the OpenAI client at http://localhost:8080. If your real stack forwards provider cache-control hints, echo the incoming cache-control header in the mock response to exercise that path.
Step 3: Inject the mock URL via environment
Never hardcode the API base in production code. Read it from OPENAI_BASE_URL. The official Python SDK respects this.
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ.get("OPENAI_API_KEY", "dummy"),
base_url=os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1")
)In conftest.py, set the env before the client is imported:
import pytest
@pytest.fixture(autouse=True)
def mock_env(monkeypatch):
monkeypatch.setenv("OPENAI_BASE_URL", "http://localhost:8080/v1")
monkeypatch.setenv("OPENAI_API_KEY", "test-key")Now any test that exercises your integration hits the mock, not the network. This is the core technique to test llm api in ci without real calls while keeping production code unchanged.
Step 4: Record real traffic once for realistic fixtures
Mocks drift. Capture a few real sessions locally and replay them in CI. Use VCR.py to serialize HTTP interactions.
import vcr
my_vcr = vcr.VCR(
cassette_library_dir="tests/fixtures/cassettes",
record_mode="once",
match_on=["method", "scheme", "host", "path", "body"],
ignore_localhost=True,
)
def test_real_shape():
with my_vcr.use_cassette("chat.yaml"):
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hi"}]
)
assert resp.choices[0].message.contentRun this once with OPENAI_BASE_URL pointed at the real provider. VCR writes chat.yaml. In CI, with the mock server disabled, VCR replays the cassette and no real call leaves the box. You still test llm api in ci without real calls because the cassette is static and version-controlled.
Step 5: Assert contract and parsing logic
A mock is only useful if you verify your client parses it. Write tests that check field extraction, token counting, and error handling.
def test_parsing():
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "x"}]
)
assert resp.object == "chat.completion"
assert resp.usage.total_tokens == 2
assert resp.choices[0].message.content == "MOCK_SUMMARY"Add a streaming test to force your parser to handle SSE frames:
def test_streaming():
chunks = []
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "x"}],
stream=True
)
for chunk in stream:
chunks.append(chunk.choices[0].delta.content or "")
assert "".join(chunks) == "MOCK"This catches bugs where your production code assumes a non-streaming shape or mishandles [DONE].
Step 6: Simulate provider errors and fallback
Real LLM providers return 429, 500, or timeout. Your code should retry or switch models. Even if a gateway provides automatic fallback when a provider is rate-limited or degraded, your client still needs to handle explicit failure.
Mock a 429:
@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
return JSONResponse(status_code=429, content={"error": {"message": "rate limit"}})Then test your retry wrapper:
def test_retry_on_429():
with pytest.raises(RateLimitError):
call_with_retry(max_retries=2)Simulate a timeout by sleeping longer than the client’s read timeout:
@app.post("/v1/chat/completions")
async def slow(request: Request):
await asyncio.sleep(10)
return JSONResponse({"id": "x"})If you implement fallback to a second model, run two mock servers on different ports and assert the second is called after the first fails. This validates routing logic without spending tokens.
Step 7: Wire into CI and verify success
Add a GitHub Actions job that starts the mock, runs pytest, and blocks network egress.
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: "3.12"}
- run: pip install fastapi uvicorn openai pytest vcrpy httpx
- run: uvicorn mock_server:app --port 8080 &
- run: pytest -q
- run: ./scripts/check-no-egress.shThe guard script fails if external LLM hosts are reachable:
#!/usr/bin/env bash
if curl -s --max-time 2 https://api.openai.com/v1/models -o /dev/null; then
echo "NETWORK LEAK DETECTED"; exit 1
fi
echo "no egress ok"How to verify success
Your suite is correct when:
- All pytest cases pass using only
localhostor loaded cassettes. - VCR cassettes are replayed without
record_mode="all"(no new real calls). - The egress guard exits zero (no external connection).
- Coverage shows your LLM client branches (parse, retry, fallback, stream) executed.
Following these steps lets you test llm api in ci without real calls reliably, cutting minutes off pipeline time and removing flaky third-party dependencies. The mock stays faithful to the contract, so regressions surface locally, not in production.