A reliable mock llm server ci pipeline removes flaky external dependencies from your test suite and lets you validate prompt logic without spending tokens. This guide shows how to build a minimal OpenAI-compatible mock, run it locally, and embed it in GitHub Actions for fully offline integration tests.
Step 1: Pin the API contract
Most LLM client libraries speak the OpenAI REST shape. Your mock must accept POST /v1/chat/completions with a JSON body and return the matching response schema, or the client SDK will throw.
Minimal request:
{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello"}],
"temperature": 0.7
}Minimal valid response:
{
"id": "chatcmpl-123",
"object": "chat.completion",
"created": 1690000000,
"model": "gpt-4o-mini",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "Hi there!"},
"finish_reason": "stop"
}
],
"usage": {"prompt_tokens": 5, "completion_tokens": 3, "total_tokens": 8}
}If you later add streaming, function calls, or seed parameters, extend the same contract. Do not invent new fields the client does not expect.
Step 2: Implement the mock server
FastAPI gives you a clean way to stand up the endpoint. The code below echoes the last user message and returns fixed token counts.
# mock_llm.py
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import time, uuid
app = FastAPI()
@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
body = await request.json()
model = body.get("model", "mock-model")
messages = body.get("messages", [])
last = messages[-1]["content"] if messages else ""
prompt_tokens = len(last.split())
resp = {
"id": f"chatcmpl-{uuid.uuid4().hex[:8]}",
"object": "chat.completion",
"created": int(time.time()),
"model": model,
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": f"Echo: {last}"},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": prompt_tokens,
"completion_tokens": 2,
"total_tokens": prompt_tokens + 2
}
}
return JSONResponse(resp)
@app.get("/v1/models")
async def models():
return {"object": "list", "data": [{"id": "mock-model", "object": "model"}]}Run it:
pip install fastapi uvicorn
uvicorn mock_llm:app --port 8000Hit it with curl to confirm:
curl -s http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"mock-model","messages":[{"role":"user","content":"test"}]}'Step 3: Add scenario-based fixtures
Echo is fine for smoke tests, but real tests need semantic responses. Map keywords in the prompt to canned fixtures.
FIXTURES = {
"summarize": "This is a concise summary.",
"translate": "Hola mundo",
"sentiment": "positive"
}
@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
body = await request.json()
model = body.get("model", "mock-model")
messages = body.get("messages", [])
text = messages[-1]["content"] if messages else ""
content = "Default mock response"
for key, val in FIXTURES.items():
if key in text.lower():
content = val
break
prompt_tokens = len(text.split())
resp = {
"id": f"chatcmpl-{uuid.uuid4().hex[:8]}",
"object": "chat.completion",
"created": int(time.time()),
"model": model,
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": content},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": prompt_tokens,
"completion_tokens": len(content.split()),
"total_tokens": prompt_tokens + len(content.split())
}
}
return JSONResponse(resp)Store fixtures as JSON files in fixtures/ if they grow large. Load them at startup with json.load.
Handling non-chat endpoints
If your app calls embeddings or completions, add those routes too. Keep the response shape copied from the provider docs.
Step 4: Containerize for reproducible CI
A Dockerfile removes “works on my machine” drift.
FROM python:3.11-slim
WORKDIR /app
RUN pip install fastapi uvicorn
COPY mock_llm.py .
CMD ["uvicorn", "mock_llm:app", "--host", "0.0.0.0", "--port", "8000"]Build and run:
docker build -t mock-llm .
docker run -p 8000:8000 mock-llmIn CI you can either build the image or just pip install and run uvicorn directly. The direct approach is faster for small repos.
Step 5: Wire your application to the mock
Point the OpenAI client at the mock with an environment variable. Never hardcode the base URL in source.
import os
from openai import OpenAI
client = OpenAI(
base_url=os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1"),
api_key=os.getenv("OPENAI_API_KEY", "dummy")
)
resp = client.chat.completions.create(
model="mock-model",
messages=[{"role": "user", "content": "summarize: long text"}]
)
print(resp.choices[0].message.content)When OPENAI_BASE_URL=http://localhost:8000/v1, the same code paths execute against the mock. This is the core of a mock llm server ci pipeline: zero code changes between test and prod, only env changes.
Step 6: Integrate into CI pipeline
GitHub Actions can start the server in the background, run tests, and tear down.
name: CI
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- run: pip install fastapi uvicorn openai pytest
- run: uvicorn mock_llm:app --port 8000 &
- run: sleep 3
- run: OPENAI_BASE_URL=http://localhost:8000/v1 pytest -qFor multi-service setups, use docker-compose with the mock as a service and depends_on. The key is that the mock is alive before the test runner starts.
Local pre-commit hook
Add a make test-local target that spins the mock and runs pytest. Engineers get the same mock llm server ci pipeline behavior before pushing.
test-local:
uvicorn mock_llm:app --port 8000 & \
sleep 2; \
OPENAI_BASE_URL=http://localhost:8000/v1 pytest -q; \
kill %1Step 7: Verify success
Write a pytest case that asserts on fixture content.
# test_mock.py
from openai import OpenAI
def test_summarize_fixture():
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
r = client.chat.completions.create(
model="mock-model",
messages=[{"role": "user", "content": "summarize: hello world"}]
)
assert r.choices[0].message.content == "This is a concise summary."
assert r.usage.total_tokens > 0
def test_default_fallback():
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
r = client.chat.completions.create(
model="mock-model",
messages=[{"role": "user", "content": "random input"}]
)
assert r.choices[0].message.content == "Default mock response"Run pytest with the server up. Green tests confirm the mock llm server ci pipeline is wired correctly.
You can also verify manually:
curl -s http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"mock-model","messages":[{"role":"user","content":"translate: hi"}]}'
# expect {"content":"Hola mundo"}Production parity notes
If your production traffic routes through n4n.ai, an OpenAI-compatible endpoint that addresses 240+ models with automatic fallback, keep the mock’s usage field and finish_reason values aligned with that gateway’s responses. That ensures client retry and token-accounting logic behaves identically in tests.
Match these details exactly:
idprefix (chatcmpl-is conventional)objectstring (chat.completion)usagewithprompt_tokens,completion_tokens,total_tokens- HTTP status codes (200 on success, 400 on malformed body)
If the real gateway streams, add an text/event-stream route to the mock and test with stream=True.
Common pitfalls
Binding to localhost in container. Uvicorn must use --host 0.0.0.0 inside Docker or the CI service cannot reach it.
Missing usage field. Some client wrappers compute cost from usage.total_tokens. Omit it and you’ll get None math errors.
Model name mismatches. Your app may request gpt-4o but mock only answers mock-model. Either accept any model in the mock or set model in tests to the mock’s name.
State leakage. If you store conversation state in the mock for multi-turn tests, reset it between cases. Stateless fixtures are simpler.
Extending the mock
Once the basic mock llm server ci pipeline is green, add:
- Latency simulation:
await asyncio.sleep(body.get("delay", 0))to test timeouts. - Error injection: return 429 or 500 on a header trigger to exercise fallback logic.
- Token limiting: truncate
contenttomax_tokensif provided.
These turn the mock from a stub into a real test harness. Build it once, ship it in CI, and your LLM integration tests will run fast and deterministic on every commit.