How to evaluate an agentic RAG pipeline

Most teams ship a retrieval-augmented generation system and call it done, but an agentic RAG pipeline adds decision loops that break in ways naive RAG never does. To evaluate agentic RAG pipeline behavior properly, you need to measure not just final answer quality but the trajectory of tool calls, retrievals, and query rewrites. This guide walks through a concrete evaluation harness you can stand up in an afternoon.

Step 1: Define the evaluation dimensions that matter for agentic loops

A static RAG eval scores faithfulness and relevance. An agentic system introduces control flow: the agent chooses to retrieve, reformulate, or call a tool. You must track those decisions or you will ship silent failures where the model confidently answers from memory instead of context.

Capture at minimum five signals:

• Retrieval precision@k for each invoked search
• Tool selection accuracy against a known-good trajectory
• Query rewrite similarity to an ideal rewrite
• Final answer faithfulness to retrieved context
• Step count and token cost per episode

Naive answer scoring hides the real bug. If the agent skipped a required retrieval and still produced a plausible answer, a faithfulness-only metric gives a false pass. To evaluate agentic RAG pipeline health you need the trajectory, not just the terminal state.

Define a metrics container so later steps stay typed.

from dataclasses import dataclass
from typing import List

@dataclass
class TrajectoryMetrics:
    episode_id: str
    retrieval_precision: float = 0.0
    tool_match: float = 0.0
    rewrite_cosine: float = 0.0
    faithfulness: float = 0.0
    steps: int = 0
    total_tokens: int = 0

Without this schema you will drown in ad-hoc JSON and cannot compare runs week over week.

Step 2: Build a deterministic replay harness from production traces

You cannot evaluate an agent against live vector stores. Index contents drift, ranking weights change, and network latency masks logic bugs. Freeze the retrieval layer so the only variable is your agent code.

Log every agent step to JSONL in production. Keep it minimal but complete:

{"episode_id":"e1","step":0,"type":"retrieve","query":"weather in Paris","docs":["doc_a","doc_b"]}
{"episode_id":"e1","step":1,"type":"tool","name":"calculator","input":"temp_c_to_f(12)"}
{"episode_id":"e1","step":2,"type":"answer","text":"12C is 53.6F."}

Replay by mocking the retriever to return the exact docs from the trace. The agent under test thinks it is hitting the real vector DB; it is not.

import json

def load_traces(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

def mock_retrieve(trace_steps, query):
    for s in trace_steps:
        if s["type"] == "retrieve" and s["query"] == query:
            return s["docs"]
    return []

Determinism lets you bisect regressions. When a prompt change drops tool_match from 1.0 to 0.7, you know exactly which commit caused it because the retrieval inputs are identical across runs.

Step 3: Instrument the agent with explicit span tags

If your agent framework does not emit structured spans, wrap the decision function yourself. A thin decorator captures the essential choice without polluting business logic.

import json

def span(step_type):
    def deco(fn):
        def wrapper(*args, **kwargs):
            result = fn(*args, **kwargs)
            print(json.dumps({"type": step_type, "io": (args, kwargs, result)}))
            return result
        return wrapper
    return deco

@span("retrieve")
def agent_retrieve(query):
    return vector_db.search(query, k=5)

Push these spans to a file in tests. You now have ground truth for tool_match and steps. For production, swap the print for an OpenTelemetry exporter, but the principle holds: every branch the agent takes must be observable.

A common mistake is logging only the final answer. The replay harness in Step 2 is useless without the intermediate queries. Instrument first, then evaluate.

Step 4: Generate a golden dataset with adversarial retrieval cases

Real users ask ambiguous multi-hop questions. Build 50–100 cases where a single retrieval fails and the agent must rewrite or call a second tool. Include distractors: a document that mentions “refund” but only for gift cards, when the question concerns service refunds.

Example entry:

{
  "episode_id": "gold_01",
  "initial_query": "How does the refund policy interact with the summer sale?",
  "ideal_rewrites": ["refund policy terms", "summer sale discount rules"],
  "expected_tools": ["retrieve", "retrieve", "answer"]
}

The evaluate agentic RAG pipeline task must catch when the agent stops early or retrieves the wrong doc. Seed the dataset from actual support tickets, not synthetic trivia. Tickets contain the messy coreference and implicit context that breaks agents.

Generate the ideal rewrites by having a strong model propose them, then have a human verify. Do not trust model-proposed rewrites as ground truth without review; they tend to over-decompose.

Step 5: Run offline trajectory evaluation

Write a pytest that replays golden episodes and compares expected tool sequence to actual. Exact match on the tool list is strict but correct for agentic systems: a missing retrieve is a defect.

def test_trajectory():
    traces = load_traces("gold.jsonl")
    for t in traces:
        agent = Agent(retriever=mock_retrieve)
        actual = agent.run(t["initial_query"])
        assert actual.tools == t["expected_tools"], f"tool mismatch {actual.tools}"

Compute tool_match as fraction of exact matches. For rewrite quality, embed the queries and compare cosine to ideal_rewrites. Use a frozen embedding model so scores are comparable across time.

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def rewrite_score(actual_q, ideal_qs, embed):
    vec = embed(actual_q)
    return max(cosine(vec, embed(iq)) for iq in ideal_qs)

Any episode with tool_match < 1.0 is a hard failure. A partial credit scheme hides regressions; keep the bar high and fix the agent.

Step 6: Score answer quality with a calibrated judge model

Faithfulness needs a language model. Use a strict rubric: answer must cite only provided docs and must answer the question. Avoid vague “is this good?” prompts; specify the scale and failure modes.

When calling a judge at scale, route through an OpenAI-compatible gateway like n4n.ai to get automatic fallback when a provider is rate-limited and per-token metering. Keep the prompt fixed and pin the judge model version.

from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")

def judge(question, answer, context):
    resp = client.chat.completions.create(
        model="anthropic/claude-3.5-sonnet",
        messages=[
            {"role":"system","content":"Score faithfulness 0-1. 1=answer fully supported by context. 0=any unsupported claim."},
            {"role":"user","content":f"Q:{question}\nA:{answer}\nC:{context}"}
        ]
    )
    return float(resp.choices[0].message.content.strip())

Run the judge three times and take the median to dampen variance. Treat judge scores below 0.8 as regressions. Do not use the same model family for agent and judge; cross-model evaluation reduces self-bias.

Step 7: Measure cost and latency per trajectory

Agentic loops multiply tokens. Record total_tokens from the completion responses and steps from spans. A pipeline that gains 5% faithfulness but triples cost is not a win for production budgets.

def aggregate(metrics_list):
    return {
        "avg_steps": sum(m.steps for m in metrics_list) / len(metrics_list),
        "avg_tokens": sum(m.total_tokens for m in metrics_list) / len(metrics_list),
        "p95_steps": sorted(m.steps for m in metrics_list)[int(0.95*len(metrics_list))],
    }

Attribute tokens per tool type. If the rewrite step consumes 40% of tokens, a smaller model for query reformulation is a clear optimization. The evaluation harness should output this breakdown, not just a single total.

Step 8: Verify success and wire into CI

Success means: all golden episodes pass tool_match == 1.0, mean faithfulness ≥ 0.85, and p95 step count under a budget you set (e.g., 6). These thresholds are starting points; tune them to your domain after two weeks of data.

Add the pytest suite to CI. Block merges on trajectory regressions.

# .github/workflows/eval.yml
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest tests/eval_agentic_rag.py --strict

Re-run the evaluate agentic RAG pipeline harness nightly on fresh production samples to catch drift. The moment retrieval precision drops because an upstream index changed, you will see it before users complain. Build the harness once, then let it guard every change to prompts, tools, or models.