RAGAS metrics explained: they are a family of reference-free and reference-based evaluation scores for retrieval-augmented generation (RAG) systems, where an LLM acts as judge to score faithfulness, answer relevancy, and context recall. These metrics quantify whether a generated answer is grounded in retrieved context, addresses the user’s question, and whether the retrieved context actually contained the required facts.
How RAGAS computes scores
RAGAS (Retrieval Augmented Generation Assessment) avoids requiring human references for most of its metrics. It prompts a separate LLM—typically a strong instruction-tuned model—to perform discrete judgment tasks. The judge model receives the question, the generated answer, and the retrieved contexts (and ground truth where needed). It returns a structured verdict that the library normalizes to a 0–1 score.
The core idea is decomposition. Instead of asking “is this answer good?”, RAGAS breaks the problem into atomic checks that are easier for an LLM to perform reliably. Each metric isolates one failure mode.
Faithfulness
Faithfulness measures the fraction of factual claims in the answer that are directly supported by the retrieved context. A faithful answer never hallucinates outside the provided documents.
The computation has two steps:
- The judge extracts individual statements from the answer.
- For each statement, it labels whether the context entails it.
from ragas.metrics import Faithfulness
from ragas.llms import LangchainLLMWrapper
faith = Faithfulness(llm=LangchainLLMWrapper(judge_llm))
score = await faith.ascore({
"answer": "Paris is the capital of France.",
"contexts": ["France is a country in Europe. Its capital is Paris."]
})
# score ~ 1.0A score of 1.0 means every claim traces to context. A score of 0.5 means half the claims are unsupported. In practice, we treat faithfulness below 0.85 as a retrain or prompt-fix signal.
The judge prompt is explicit: “Extract statements. For each, answer YES if context supports it.” This narrow task outperforms holistic grading because the LLM does not need to weigh style or completeness.
Answer relevancy
Answer relevancy (sometimes called relevance) checks whether the answer actually addresses the question, independent of factual correctness. RAGAS computes this by reversing the task: the judge generates plausible questions from the answer, then measures semantic similarity between those synthetic questions and the original question.
If the answer is off-topic, the generated questions will diverge, dropping the score.
from ragas.metrics import AnswerRelevancy
ar = AnswerRelevancy(llm=LangchainLLMWrapper(judge_llm))
score = await ar.ascore({
"question": "What is the capital of France?",
"answer": "The Eiffel Tower is 330 meters tall.",
"contexts": ["The Eiffel Tower is in Paris."]
})
# low score: answer is irrelevant to the questionThe metric uses an embedding model to compute cosine similarity between the original question embedding and the mean of generated question embeddings. That makes it sensitive to semantic drift but blind to factual errors.
Context recall
Context recall is the only one of the three that requires ground-truth references. It measures whether the retrieved context includes the facts needed to answer the question correctly. The judge compares the ground-truth answer against the concatenated retrieved passages and counts how many required facts were present.
from ragas.metrics import ContextRecall
cr = ContextRecall(llm=LangchainLLMWrapper(judge_llm))
score = await cr.ascore({
"question": "What is the capital of France?",
"ground_truth": "Paris",
"contexts": ["Lyon is a city in France."]
})
# score 0.0: context missed the needed factWithout ground truth, you cannot compute this score. Teams often skip it early, then add a labeled set once they have sampled production queries.
Why these metrics matter in production
A RAG system fails silently. The retriever can return junk, the generator can ignore context, and the user still gets a plausible sentence. Traditional unit tests on string equality catch none of this.
RAGAS metrics explained in practice give you a regression harness. You snapshot scores on a golden set of queries. After a vector DB reindex or prompt change, you re-run. If faithfulness drops from 0.92 to 0.74, you have a concrete signal before users complain.
They also decouple concerns. Low context recall points at the retriever. Low faithfulness with high context recall points at the generator ignoring instructions. Low answer relevancy points at query understanding or prompt formatting.
A concrete evaluation run
Assume you have a small evaluation dataset. Using the ragas library with an OpenAI-compatible backend:
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset
data = Dataset.from_dict({
"question": [
"What is the capital of France?",
"Who wrote Hamlet?"
],
"answer": [
"Paris is the capital of France.",
"Hamlet was written by William Shakespeare."
],
"contexts": [
["France is a country in Europe. Its capital is Paris."],
["Shakespeare authored many plays, including Hamlet."]
],
"ground_truth": [
"Paris",
"William Shakespeare"
]
})
result = evaluate(
data,
metrics=[faithfulness, answer_relevancy, context_recall],
llm=judge_llm,
embeddings=embed_model
)
print(result)The output is a dict with mean scores per metric. Wire this into CI as an artifact.
When running these judgments at scale, route the judge calls through an OpenAI-compatible gateway like n4n.ai that exposes 240+ models with automatic fallback when a provider is rate-limited. Batch scoring across thousands of rows will otherwise trip provider quotas mid-run.
Common misconceptions
High faithfulness means the answer is correct
False. Faithfulness only checks grounding in retrieved context. If the retriever pulled a poisoned document stating “Paris is in Germany”, a faithful answer repeating that is scored 1.0. You still need context precision and external truth checks.
Answer relevancy is the same as correctness
No. An answer can be perfectly on-topic but factually wrong. Relevancy uses semantic similarity of reversed questions; it never validates facts.
RAGAS is fully reference-free
Only faithfulness and answer relevancy are. Context recall requires ground_truth. If your eval set lacks references, you cannot compute recall.
Scores are deterministic
LLM judges are stochastic. Set temperature to 0 for the judge and run multiple seeds if you need stable gates. A 0.01 swing is noise.
One aggregate score is enough
Engineers often collapse everything into a single “ragas score”. That hides failures. Track the three metrics separately; they diagnose different components.
Setting thresholds that aren’t arbitrary
Do not pick 0.9 because it feels good. Sample 100 historical queries, label them manually as pass/fail, then compute the ROC curve of each metric against your labels. Choose the cutoff where false negatives hurt least. For faithfulness, most teams land between 0.8 and 0.9 once judge quality is decent.
Log the distribution, not just the mean. A mean of 0.9 with half the rows at 0.4 indicates a sporadic retrieval bug that averages hide.
Observability wiring
Pipe the per-row scores into your existing metrics stack (Prometheus, Grafana, or a managed trace system). Tag them with the retriever version and prompt hash. When a deploy shifts context recall, you want to correlate it with the exact vector index commit.
Per-token usage metering on judge calls matters because evaluation can cost more than inference if you loop daily over 10k rows. Cache judge inputs and reuse across experiments.
Cost and caching notes
The judge LLM runs once per answer for faithfulness, plus extra generations for relevancy question synthesis. At scale, that is three to five completions per evaluated row. Use provider cache-control hints so identical context blocks are not re-scored. If your gateway honors client routing directives, pin the judge to a cheaper model that still meets your correlation bar.
RAGAS metrics explained here should give you enough to instrument your pipeline today. Start with a 50-row golden set, measure faithfulness and answer relevancy weekly, add context recall once you label ground truths. The visibility pays for itself at the first silent retrieval regression.