Most RAG systems rot silently: a chunking change breaks retrieval, an embedding model swap skews similarity, a prompt edit degrades answers. A solid CI/CD pipeline for RAG applications catches these regressions before they hit production by treating indexes, retrievers, and generation as testable artifacts. This tutorial builds one with GitHub Actions, Python, and a deterministic eval harness you can run locally and in CI.
Traditional CI assumes code is the only variable. In RAG, the data, the embedding model, and the generation model all shift underneath you. The pipeline we build separates those concerns into fast unit tests, slower retrieval integration tests, and a gated LLM eval.
Prerequisites
- Python 3.11+ and a GitHub repository.
pip install chromadb sentence-transformers openai pytest.- An OpenAI-compatible API key (any provider or gateway).
- Basic familiarity with pytest and GitHub Actions YAML.
We will not cover vector DB hosting or deployment to production—only the automated quality gate that should run on every pull request.
Project layout
rag-demo/
├── rag.py
├── tests/
│ ├── test_chunking.py
│ ├── test_retrieval.py
│ └── test_eval.py
├── eval.py
└── .github/workflows/ci.ymlStep 1: A minimal RAG module
rag.py implements chunking, embedding, and a query path. We use sentence-transformers for local embeddings and Chroma for the index.
import chromadb
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
client = chromadb.Client()
coll = client.create_collection("docs")
def chunk_text(text, size=200, overlap=20):
words = text.split()
step = size - overlap
return [" ".join(words[i:i+size]) for i in range(0, len(words), step)]
def build_index(docs):
chunks = [c for d in docs for c in chunk_text(d)]
emb = model.encode(chunks).tolist()
coll.add(ids=[str(i) for i in range(len(chunks))],
embeddings=emb, documents=chunks)
return chunks
def retrieve(query, k=3):
q_emb = model.encode([query]).tolist()
return coll.query(query_embeddings=q_emb, n_results=k)This is intentionally small. The point is to have pure functions and a clear boundary for tests. In a larger app, build_index would point at a persistent store and accept a configurable embedding model name.
Step 2: Test chunking and embedding
Chunking regressions are common. A simple unit test locks behavior so a later “optimization” can’t silently merge unrelated sections.
from rag import chunk_text
def test_chunk_count():
text = "word " * 420
chunks = chunk_text(text, size=200, overlap=20)
assert len(chunks) == 3
def test_overlap_preserved():
chunks = chunk_text("a b c d e f g h i j", size=4, overlap=2)
assert chunks[0].split()[-2:] == chunks[1].split()[:2]Run pytest tests/test_chunking.py -q. Expected output:
.. [100%]
2 passed in 0.31sThese tests run in milliseconds and need no network. They belong in the same job as your lint step.
Step 3: Retrieval integration test
Retrieval is where RAG lives or dies. Index a known document and assert the top hit contains the source phrase. This catches embedding dimension mismatches and Chroma schema drift.
from rag import build_index, retrieve
def test_retrieval_hits():
build_index(["The capital of France is Paris. Snails are tasty."])
res = retrieve("What is the capital of France?", k=1)
docs = res["documents"][0]
assert any("Paris" in d for d in docs)Output:
. [100%]
1 passed in 1.12sIf someone changes chunk_text to drop overlap, this test may still pass, but the next step catches semantic drift that pure lexical checks miss.
Step 4: Generation eval with an LLM judge
A retrieval test does not prove the answer is correct. We add eval.py that queries a model and uses a second model as a judge. Point the OpenAI client at n4n.ai’s OpenAI-compatible endpoint to run the judge against multiple models without code changes; it meters per-token usage and forwards cache hints so repeated eval prompts stay cheap.
import os, json
from openai import OpenAI
from rag import build_index, retrieve
client = OpenAI(base_url=os.environ["OPENAI_BASE_URL"],
api_key=os.environ["OPENAI_API_KEY"])
def generate(question):
build_index(["The capital of France is Paris. The Eiffel Tower is in Paris."])
ctx = retrieve(question, k=2)["documents"][0]
prompt = f"Context: {ctx}\nQuestion: {question}\nAnswer:"
r = client.chat.completions.create(
model=os.environ.get("MODEL", "gpt-4o-mini"),
temperature=0,
messages=[{"role": "user", "content": prompt}])
return r.choices[0].message.content
def judge(question, answer):
sys = "You are a strict grader. Reply JSON: {\"pass\": bool, \"reason\": str}"
r = client.chat.completions.create(
model=os.environ.get("JUDGE_MODEL", "gpt-4o-mini"),
temperature=0,
messages=[{"role": "system", "content": sys},
{"role": "user", "content": f"Q: {question} A: {answer}"}])
return json.loads(r.choices[0].message.content)
if __name__ == "__main__":
q = "What is the capital of France?"
ans = generate(q)
verdict = judge(q, ans)
print(json.dumps({"answer": ans, "verdict": verdict}))Why temperature zero
LLM judges are flaky if they sample. Setting temperature=0 makes the grader deterministic across CI runs. If your gateway supports cache-control, the judge prompt is identical per question, so it gets served from cache after the first call.
Run OPENAI_BASE_URL=https://api.n4n.ai/v1 OPENAI_API_KEY=sk-... python eval.py. Sample output:
{"answer": "The capital of France is Paris.", "verdict": {"pass": true, "reason": "Answer matches context"}}If verdict.pass is false, the CI job should fail. Wrap the script so a non-pass exits 1.
Step 5: Assemble the CI/CD pipeline for RAG applications
The GitHub Actions workflow installs deps, runs unit/integration tests, then runs the eval gate. This CI/CD pipeline for RAG applications runs on every pull request and blocks merges on regression.
name: rag-ci
on: [pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.11" }
- run: pip install chromadb sentence-transformers openai pytest
- run: pytest tests/ -q
- name: eval gate
env:
OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
MODEL: gpt-4o-mini
JUDGE_MODEL: gpt-4o-mini
run: |
python eval.py | tee eval.out
grep -q '"pass": true' eval.out || exit 1Caching in CI
Installing sentence-transformers downloads an 80MB model. Add actions/cache for pip and the HuggingFace cache to keep job time under a minute. The eval network call is the long pole; run it only after the fast tests pass.
Step 6: Verify the pipeline
Push a branch. GitHub Actions shows:
pytest tests/ -q
...
3 passed in 1.45s
python eval.py | tee eval.out
{"answer": "The capital of France is Paris.", "verdict": {"pass": true, "reason": "Answer matches context"}}If you deliberately break chunk_text to return empty strings, the retrieval test fails and the job stops before the eval. If you swap the embedding model to a random projector, retrieval may still return something but the judge will likely flag the answer as unsupported.
Extending the gate
For real systems, replace the single-question eval with a bundled dataset of (question, expected_fact) pairs. Store embeddings in a persistent Chroma instance rather than an in-memory client. Cache the judge model responses with provider cache-control headers to cut cost. The core pattern stays: chunk/embed tests, retrieval assertions, and an LLM judge in the same CI/CD pipeline for RAG applications you already trust for regular software.
A good RAG pipeline is not a notebook; it is a tested component. Wire these checks into your repo today and your future prompt edits will thank you.