LangChain RAG evaluation with RAGAS and n4n.ai

Evaluating a RAG pipeline is not optional — it is the only way to know whether your retrieval and generation components actually work together. This guide walks through langchain rag evaluation ragas from a blank project to a repeatable CI job, using synthetic data so you can iterate before you have production logs.

Step 1: Set up the environment

Create a virtual environment and install the minimal dependencies. RAGAS pulls in LangChain, so you do not need to install LangChain separately unless you want a specific version.

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install "ragas[langchain]" langchain-openai langchain-community pandas

If you use n4n.ai as your inference gateway, set the base URL and key once and the OpenAI-compatible client will route through 240+ models with automatic fallback. The code below works unchanged; only the environment variables differ.

export OPENAI_API_KEY="your-n4n-key"
export OPENAI_BASE_URL="https://api.n4n.ai/v1"

Verify the install:

# verify.py
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
print("RAGAS version:", evaluate.__module__)

Run python verify.py — no errors means you are ready.

Step 2: Build a minimal RAG chain

You need a runnable LangChain chain that returns both an answer and the retrieved contexts. The following example uses an in-memory vector store so the tutorial runs without external services. Swap the retriever for Pinecone, Weaviate, or PGVector in production.

# rag_chain.py
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# 1. Tiny corpus
docs = [
    Document(page_content="The n4n.ai gateway routes requests to 240+ models with automatic fallback.", metadata={"source": "docs/overview.md"}),
    Document(page_content="Per-token usage metering is exposed via response headers.", metadata={"source": "docs/metering.md"}),
    Document(page_content="Client routing directives are honored; provider cache-control hints are forwarded.", metadata={"source": "docs/routing.md"}),
]

# 2. Vector store + retriever
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = FAISS.from_documents(docs, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

# 3. Prompt
prompt = ChatPromptTemplate.from_template("""Answer the question using only the context below.
If the answer is not in the context, say you don't know.

Context:
{context}

Question: {question}
Answer:""")

# 4. LLM
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# 5. Chain that returns contexts + answer
def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Helper for RAGAS: returns dict with answer and contexts
def run_rag(question: str) -> dict:
    retrieved = retriever.invoke(question)
    answer = rag_chain.invoke(question)
    return {
        "question": question,
        "answer": answer,
        "contexts": [d.page_content for d in retrieved],
    }

Test it:

# test_chain.py
from rag_chain import run_rag
result = run_rag("How many models does the gateway support?")
print(result["answer"])
print("--- contexts ---")
for c in result["contexts"]:
    print(c[:80], "...")

You should see a concise answer grounded in the first document.

Step 3: Generate a synthetic evaluation dataset

RAGAS works best with a dataset that covers your expected query distribution. The TestsetGenerator creates question–context–ground_truth triples from your corpus. This is the fastest way to start langchain rag evaluation ragas before you have real user traffic.

# generate_dataset.py
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document
import pandas as pd

# Reuse the same corpus
docs = [
    Document(page_content="The n4n.ai gateway routes requests to 240+ models with automatic fallback.", metadata={"source": "docs/overview.md"}),
    Document(page_content="Per-token usage metering is exposed via response headers.", metadata={"source": "docs/metering.md"}),
    Document(page_content="Client routing directives are honored; provider cache-control hints are forwarded.", metadata={"source": "docs/routing.md"}),
]

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = FAISS.from_documents(docs, embeddings)

# Generator uses a critic LLM and an embedding model
generator_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
critic_llm = ChatOpenAI(model="gpt-4o", temperature=0)

generator = TestsetGenerator.from_langchain(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings=embeddings,
)

# Distribution: 50% simple, 25% reasoning, 25% multi-context
testset = generator.generate_with_langchain_docs(
    docs,
    test_size=20,
    distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},
)

# Convert to pandas and save
df = testset.to_pandas()
df.to_csv("eval_dataset.csv", index=False)
print(df[["question", "ground_truth"]].head())

Run python generate_dataset.py. Open eval_dataset.csv — you should see 20 rows with question, ground_truth, contexts, and evolution_type. If the file is empty, check your API key and quota.

Step 4: Run the RAG pipeline over the dataset

Now execute your RAG chain on every question in the dataset and collect answers and retrieved contexts. RAGAS expects a Dataset object with specific column names.

# run_evaluation.py
from datasets import Dataset
from rag_chain import run_rag
import pandas as pd

df = pd.read_csv("eval_dataset.csv")

results = []
for _, row in df.iterrows():
    q = row["question"]
    out = run_rag(q)
    results.append({
        "question": q,
        "answer": out["answer"],
        "contexts": out["contexts"],
        "ground_truth": row["ground_truth"],
    })

eval_dataset = Dataset.from_list(results)
eval_dataset.save_to_disk("ragas_eval_dataset")
print(f"Saved {len(eval_dataset)} examples")

Verify the schema:

# check_schema.py
from datasets import load_from_disk
ds = load_from_disk("ragas_eval_dataset")
print(ds.column_names)
# Expected: ['question', 'answer', 'contexts', 'ground_truth']

Step 5: Choose and run metrics

RAGAS ships with retrieval and generation metrics. For a first pass, run the four core metrics:

context_precision — of the retrieved chunks, how many are relevant?
context_recall — does the retrieved set cover the ground truth?
faithfulness — does the answer stay grounded in the retrieved contexts?
answer_relevancy — does the answer address the question?

# evaluate_metrics.py
from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from datasets import load_from_disk

dataset = load_from_disk("ragas_eval_dataset")

metrics = [context_precision, context_recall, faithfulness, answer_relevancy]

result = evaluate(dataset, metrics=metrics)
df_result = result.to_pandas()
df_result.to_csv("ragas_scores.csv", index=False)
print(df_result[["context_precision", "context_recall", "faithfulness", "answer_relevancy"]].describe())

The output CSV contains a score per example plus aggregate statistics. A healthy baseline on synthetic data:

context_precision > 0.7
context_recall > 0.6
faithfulness > 0.8
answer_relevancy > 0.7

If faithfulness is low, your prompt lets the model hallucinate. If context_recall is low, increase k or improve the retriever.

Step 6: Inspect failures manually

Aggregate scores hide regression patterns. Pull the lowest-scoring examples and read them.

# inspect_failures.py
import pandas as pd

df = pd.read_csv("ragas_scores.csv")
# Bottom 3 by faithfulness
worst = df.nsmallest(3, "faithfulness")[["question", "answer", "contexts", "faithfulness", "ground_truth"]]
for _, row in worst.iterrows():
    print(f"Q: {row['question']}")
    print(f"Faithfulness: {row['faithfulness']:.2f}")
    print(f"Answer: {row['answer']}")
    print(f"Ground truth: {row['ground_truth']}")
    print("Contexts:")
    for c in eval(row["contexts"]):
        print(f"  - {c[:100]}")
    print("---")

Common failure modes:

Retriever misses key entity → add synonyms to corpus or use hybrid search.
Model ignores context → tighten prompt, add “only use context” instruction, lower temperature.
Answer is correct but not in ground truth → ground truth is incomplete; update the dataset.

Step 7: Iterate on the retriever and prompt

Evaluation is a loop. Change one thing, re-run Step 4–6, compare scores.

Increase retrieval depth

# In rag_chain.py
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})  # was 2

Switch to hybrid search (BM25 + vector)

# hybrid_retriever.py
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever

bm25 = BM25Retriever.from_documents(docs)
bm25.k = 2
ensemble = EnsembleRetriever(retrievers=[bm25, retriever], weights=[0.5, 0.5])

Replace retriever in the chain with ensemble. Re-run evaluation. Expect context_recall to rise; watch context_precision for noise.

Prompt hardening

prompt = ChatPromptTemplate.from_template("""You are a precise technical assistant.
Answer the question using ONLY the provided context.
If the context does not contain the answer, respond exactly: "I don't know based on the provided context."

Context:
{context}

Question: {question}
Answer:""")

Re-run. Faithfulness should improve; answer_relevancy may dip slightly if the model becomes overly cautious.

Step 8: Persist the evaluation dataset as a regression suite

Synthetic data is a starting point. As real queries arrive, add them to the dataset with verified ground truth. Store the CSV in version control.

git add eval_dataset.csv ragas_scores.csv
git commit -m "Add baseline RAGAS evaluation dataset and scores"

When you change the retriever, prompt, or model, run the evaluation again and diff the scores:

python evaluate_metrics.py
git diff ragas_scores.csv

A drop in any metric blocks the PR.

Step 9: Wire into CI

Add a GitHub Actions job that runs on every push to main and on pull requests. The job installs dependencies, runs the pipeline, and fails if faithfulness drops below a threshold.

# .github/workflows/rag-eval.yml
name: RAG Evaluation
on:
  push:
    branches: [main]
  pull_request:

jobs:
  evaluate:
    runs-on: ubuntu-latest
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - run: python generate_dataset.py
      - run: python run_evaluation.py
      - run: python evaluate_metrics.py
      - name: Check faithfulness threshold
        run: |
          python -c "
          import pandas as pd
          df = pd.read_csv('ragas_scores.csv')
          mean_faith = df['faithfulness'].mean()
          print(f'Mean faithfulness: {mean_faith:.3f}')
          if mean_faith < 0.75:
              print('::error::Faithfulness below threshold')
              exit(1)
          "

Add requirements.txt:

ragas[langchain]==0.2.5
langchain-openai==0.1.25
langchain-community==0.2.10
pandas==2.2.0
datasets==2.18.0
faiss-cpu==1.8.0

Pin versions. RAGAS and LangChain move fast; unpinned dependencies will break your CI.

Step 10: Extend with custom metrics

The built-in metrics cover the basics. For domain-specific needs — citation accuracy, numeric precision, policy compliance — write a custom metric by subclassing MetricWithLLM.

# custom_metrics.py
from ragas.metrics import MetricWithLLM
from ragas.prompt import Prompt
from pydantic import Field

class CitationAccuracy(MetricWithLLM):
    name: str = "citation_accuracy"
    evaluation_mode: str = "qac"
    prompt: Prompt = Field(default_factory=lambda: Prompt(
        name="citation_accuracy",
        instruction="""Does the answer cite the provided contexts correctly?
Score 1 if every factual claim in the answer is supported by a cited context.
Score 0 if any claim is unsupported or misattributed.
Return only the numeric score.""",
        input_keys=["answer", "contexts"],
    ))

    def _score(self, row, callbacks=None):
        return self._evaluate_row(row, callbacks)

from custom_metrics import CitationAccuracy
metrics.append(CitationAccuracy())

Run evaluation again. The new column appears in ragas_scores.csv.

Verification checklist

Before you consider the evaluation pipeline done, confirm each item:

python verify.py prints RAGAS version without error.
python test_chain.py returns a grounded answer for a known question.
eval_dataset.csv exists with ≥20 rows and columns question, ground_truth, contexts.
ragas_scores.csv contains four metric columns plus per-row scores.
python inspect_failures.py outputs readable failure.py` surfaces actionable failure cases.
CI job passes on main and fails when you intentionally degrade the prompt.
requirements.txt is pinned and committed.

What to do next

Replace the in-memory FAISS store with your production vector database and re-run the full suite.
Swap gpt-4o-mini for a smaller model (e.g., gpt-4o-mini → gpt-3.5-turbo) and watch latency vs. faithfulness trade-offs.
Add a nightly job that regenerates the synthetic dataset with a larger test_size to catch distribution drift.
Feed production logs into the dataset: sample 50 real queries per week, have a domain expert label ground truth, append to eval_dataset.csv.

LangChain RAG evaluation with RAGAS is not a one-time script — it is a regression suite for your retrieval–generation contract. Treat it like unit tests: run it on every change, fail the build on regressions, and expand coverage as the product evolves.