Ragas metrics for RAG are a set of reference-free evaluation scores that measure how well a retrieval-augmented generation pipeline fetches relevant context and produces faithful, on-topic answers. Context precision and answer relevancy are two of the most used signals: the former scores whether retrieved chunks are ranked by usefulness, the latter scores whether the generated answer addresses the user query.
Why Ragas metrics for RAG exist
Classic IR evaluation (e.g., MAP, nDCG) needs graded relevance judgments from humans. In RAG systems, the “right” context is often ambiguous and shifts as you tweak chunking or embedding models. Ragas metrics for RAG replace static labels with an LLM judge that reads the query, the retrieved passages, and the generated answer, then emits a 0–1 score. This makes evaluation reproducible in CI without a permanent annotation team.
The trade-off is clear: you swap human ground truth for a stochastic judge. That is acceptable when you treat scores as relative signals across pipeline versions, not as absolute quality certificates.
Context precision: definition and mechanics
Context precision answers one question: given the ordered list of retrieved chunks, do relevant chunks appear early? A RAG retriever that returns the gold document at position 8 but fills positions 1–7 with noise has terrible context precision, even if the answer is eventually correct.
How the LLM judge labels relevance
For each retrieved context c_i (in rank order), the judge prompt asks: “Is this context necessary to answer the question?” The model returns a binary yes/no. Ragas then computes precision at each rank k only where the item is relevant:
precision@k = (number of relevant items in positions 1..k) / k
context_precision = sum over k where relevant_k=1 of precision@k / total_relevantIf the first three chunks are relevant and the fourth is not, precision@1=1, @2=1, @3=1, and the irrelevant fourth doesn’t enter the numerator. The score rewards ranking, not just recall.
Running it
from ragas import evaluate
from ragas.metrics import context_precision
from datasets import Dataset
data = Dataset.from_dict({
"question": ["What is the refund window for Pro plans?"],
"contexts": [[
"Pro plans support SSO and audit logs.",
"Refunds for Pro plans are issued within 14 days of charge.",
"The company was founded in 2019."
]],
"answer": ["Refunds for Pro plans happen within 14 days."]
})
result = evaluate(data, metrics=[context_precision])
print(result[context_precision.name])The judge LLM (default: a hosted model) will mark context 2 as relevant and contexts 1 and 3 as not. Precision@2 = 1/2 = 0.5, but only context 2 is relevant, so the final score is 0.5. Swap the order so the refund sentence is first, and the score jumps to 1.0.
Answer relevancy: not the same as relevance
Engineers new to Ragas metrics for RAG often conflate answer relevancy with faithfulness or with context precision. Answer relevancy specifically measures topical alignment between the user question and the generated answer.
Pseudo-question generation
Ragas answer relevancy works by asking the LLM to generate n (default 3) plausible questions that the given answer could respond to. It embeds the original question and those pseudo-questions, then averages the cosine similarity. A high score means the answer is semantically on-target; a low score means the model drifted into a tangential essay.
from ragas.metrics import answer_relevancy
# answer_relevancy requires an embedding model and an LLM in the ragas config
result = evaluate(data, metrics=[answer_relevancy])This catches the classic failure where the model says “That’s a great question, here is general background…” and never answers.
A concrete evaluation example
Suppose you run a support bot over a knowledge base. You assemble a golden set of 50 historical tickets. Two rows:
[
{
"question": "Can I use my license on two laptops?",
"contexts": [
"Our EULA permits activation on up to 3 devices.",
"Support hours are 9-5 PT.",
"The desktop app requires Windows 10."
],
"answer": "Yes, you can activate on up to 3 devices."
},
{
"question": "How do I export my data?",
"contexts": [
"Data export is available under Settings > Privacy.",
"We host on AWS us-east-1.",
"The API rate limit is 100 req/min."
],
"answer": "Export is in Settings > Privacy."
}
]Run both context precision and answer relevancy:
from ragas.metrics import context_precision, answer_relevancy
results = evaluate(dataset, metrics=[context_precision, answer_relevancy])
print(results)Output might look like:
{
"context_precision": 0.83,
"answer_relevancy": 0.91
}The 0.83 tells you one retrieved chunk in some rows was irrelevant but ranked before a relevant one. The 0.91 says answers stay on topic. If you later change the embedding model and context_precision drops to 0.61, you have a regression in retrieval ranking even if answer_relevancy stays flat.
Common misconceptions about Ragas metrics for RAG
Precision is not recall
Context precision ignores whether you retrieved all relevant docs. A retriever that returns exactly one perfect chunk at rank 1 scores 1.0 on precision but may miss three other critical passages. Pair it with context recall (or faithfulness) before declaring victory.
The LLM judge is not an oracle
Judge models exhibit position bias (preferring early contexts) and verbosity bias (rewarding long answers). Ragas mitigates some of this with prompt design, but you should still spot-check 20–30 low-score cases by hand each release. Treat score deltas > 0.05 as signal; deltas < 0.02 are noise.
High answer relevancy does not imply correct facts
An answer can be perfectly on-topic and completely wrong. “You can export data via the Delete Account button” is topically aligned with the export question but harmful. That is why faithfulness (alignment to retrieved context) is a separate metric you must run.
Thresholds are pipeline-specific
A 0.9 context precision bar for a legal RAG system with dense documents is unrealistic if your chunker splits sentences aggressively. Calibrate thresholds against historical human-approved launches, not against a blog post.
Running evaluations in production
Wire Ragas into a nightly job that pulls a fixed golden dataset and fails the build if context precision drops below your baseline minus a margin. Keep the dataset small (50–200 items) so the judge LLM cost stays trivial.
The judge call is just another LLM completion. Pointing it at a gateway like n4n.ai gives you one OpenAI-compatible endpoint that addresses 240+ models with automatic fallback when a provider is rate-limited, so nightly evals don’t fail on upstream 429s. You keep the same openai client config and add a base_url.
import os
os.environ["OPENAI_API_KEY"] = "sk-...",
os.environ["OPENAI_BASE_URL"] = "https://api.n4n.ai/v1"
# ragas will use the openai client under the hood for the judgeMinimal CI snippet
pip install ragas datasets
python eval_ragas.py --dataset golden.json --metric context_precision --min-score 0.75If the script exits non-zero, the pipeline change is blocked until a human reviews the diff.
Ragas metrics for RAG give you cheap, repeatable signals about retrieval ranking and answer focus. They are not a substitute for end-to-end human review, but they make regressions visible the moment a chunking or embedding change ships.