Context relevance scoring RAG is the process of quantifying how well retrieved passages align with the user’s query before those passages are fed to a generator. It produces a numeric or categorical signal that tells you whether your retriever surfaced useful context, independent of the final answer quality.
What context relevance scoring actually measures
Retrieval-augmented generation splits into two failure domains: retrieval and generation. Context relevance scoring RAG isolates the first. It asks a single question: given the query, would a human judge consider this chunk worth reading to answer it?
The signal is not answer correctness. A passage can be perfectly relevant and still lead to a wrong final answer because the generator misread it. Conversely, a generated answer can be correct while citing irrelevant context through lucky reasoning. Scoring relevance decouples these concerns so you can localize regressions.
In practice the score is either a float in [0,1], a 1–5 rating, or a binary label. The granularity depends on how you compute it and what your monitoring dashboard can display without lying to viewers.
How it works in practice
There are two families of scorers that ship in real systems: overlap-based and model-based. Most teams start with the former and migrate to the latter once volume justifies the cost.
Lexical and embedding overlap
The cheapest method computes similarity between the query embedding and each retrieved chunk embedding. Cosine similarity above a threshold is “relevant.” This catches obvious mismatches but fails on paraphrases where vocabulary diverges.
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
def relevance_score(query: str, chunk: str) -> float:
q_emb = model.encode(query, normalize_embeddings=True)
c_emb = model.encode(chunk, normalize_embeddings=True)
return float(np.dot(q_emb, c_emb))A similarity of 0.82 might be relevant for factual queries; 0.31 is almost certainly noise. The threshold is domain-specific. Set it from a labeled sample, not intuition.
Lexical overlap (token Jaccard, BM25 match) is even noisier but requires no model. It is useful as a sanity check that your vector store isn’t returning empty or duplicated rows.
LLM-as-judge scoring
For nuanced queries, an LLM grades the pair. You pass the query and the chunk, ask for a strict relevance verdict, and parse the response. This is slower and costs tokens, but it captures semantic fit that embeddings miss.
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY") # OpenAI-compatible, fallback built in
PROMPT = """You are a strict relevance grader.
Query: {query}
Passage: {passage}
Rate relevance 1-5 where 1 is unrelated and 5 is directly answers query.
Return only the integer."""
def llm_relevance(query: str, passage: str) -> int:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": PROMPT.format(query=query, passage=passage)}],
temperature=0,
)
return int(resp.choices[0].message.content.strip())When you call an LLM to compute context relevance scoring RAG metrics, routing through an OpenAI-compatible gateway such as n4n.ai gives you automatic fallback across providers so a rate limit doesn’t stall your nightly eval job. The per-token metering also keeps the cost of grading visible per pipeline run.
Why it matters for pipeline monitoring
Without context relevance scoring RAG, you are blind to retrieval failures until users complain. Retrieval quality drifts constantly: embedding model updates, chunk-size tweaks, vector index rebalancing, or upstream document changes all shift what surfaces.
A monitoring graph of mean relevance per query type catches:
- Index corruption (sudden drop to near zero)
- Embedding drift after a model swap
- Partial coverage gaps (low scores only on “billing” queries)
You can alert on the 5th percentile rather than the mean. If even 5% of retrieved contexts score below 0.2, certain user segments are getting garbage context and will churn.
Context relevance also gates cost. If 30% of retrieved chunks are irrelevant, you are paying to embed, store, and generate against tokens that add latency and hallucination risk. Cutting them improves both margin and accuracy.
A concrete implementation example
Assume a support bot over a documentation corpus. We evaluate the retriever nightly on 200 held-out queries with known relevant doc IDs.
Dataset and retriever setup
{
"query": "How do I rotate API keys?",
"retrieved": [
{"id": "doc-882", "text": "To rotate keys, open settings > security > rotate..."},
{"id": "doc-101", "text": "Billing cycles close on the last day of month."}
],
"expected_ids": ["doc-882"]
}Scoring code
We blend embedding similarity with an LLM judge for the top-3 hits, then log the fraction of queries where at least one relevant chunk appears in the top-3 (recall@3) and the average judge score.
import json, numpy as np
from sklearn.metrics import recall_score
def evaluate(run_path: str):
scores, y_true, y_pred = [], [], []
for line in open(run_path):
item = json.loads(line)
q = item["query"]
for doc in item["retrieved"][:3]:
s = llm_relevance(q, doc["text"])
scores.append(s)
rel = int(doc["id"] in item["expected_ids"])
y_true.append(1 if rel else 0)
y_pred.append(1 if s >= 3 else 0)
print("mean_judge_score", np.mean(scores))
print("recall@3", recall_score(y_true, y_pred, zero_division=0))
evaluate("nightly_retrieval.jsonl")The output tells you two things: the average quality of what you retrieve, and whether the right doc is present at all. Both are needed; a high average score with low recall means you retrieve four good chunks but never the one the query needed.
This implements a minimal context relevance scoring RAG evaluator that fits in a single cron job. Expand it with per-topic breakdowns once the baseline is stable.
Common misconceptions
Relevance equals answer correctness
Engineers new to RAG pipeline observability often treat a high relevance score as proof the bot works. It is not. The generator can ignore the context, contradict it, or cite it wrongly. Relevance scoring monitors the retriever only; pair it with answer-faithfulness eval to cover the full path.
High recall is enough
Recall@k looks reassuring until you notice the generator picks the wrong chunk. If you retrieve ten passages and one is relevant, recall@10 is 100% but the signal-to-noise ratio is terrible. Always track precision or mean judge score alongside recall, or you will mask cost and latency bloat.
The score is stable across models
Embedding-based relevance thresholds calibrated on all-MiniLM-L6-v2 do not transfer to text-embedding-3-large. LLM judges change behavior between model versions and temperature settings. Re-baseline your thresholds whenever you swap a scorer. Treat the number as relative to the scoring method, not an absolute ground truth.
Binary labels are sufficient for tuning
A binary “relevant / not” hides gradations that matter for ranking. A chunk that partially answers the query should outrank one that merely shares keywords. Use ordinal or continuous scores when optimizing the retriever; reserve binary for hard alerting rules.
Operational notes
Store the raw scores, not just aggregates. When a query cluster degrades, you need the per-chunk distribution to see whether it is a single bad embedding or a systemic miss. Pipe the scores into the same tracing system as your generation logs so a support ticket can be traced from answer back to retrieved context.
Context relevance scoring RAG is not a one-time benchmark. It is a continuous sensor. The teams that get value from it wire it into the same CI that deploys retriever changes, so a relevance regression blocks the rollout exactly like a unit test failure.