Poor answers from a retrieval-augmented generation system almost always trace back to the retriever, not the LLM. Debugging retrieval quality RAG means treating the vector search as a measurable subsystem: you need labeled queries, metric baselines, and a way to isolate where relevant context gets lost before generation.
Step 1: Build a labeled evaluation set
You cannot improve what you cannot measure. Start by collecting 50–200 real or representative queries and the document IDs that should be retrieved for each. Pull these from support tickets, search logs, or synthetic questions written by someone who knows the domain.
Store them in a simple JSONL file:
{"query": "How do I rotate API keys?", "relevant_ids": ["doc_123", "doc_124"]}
{"query": "What is the rate limit for batch endpoints?", "relevant_ids": ["doc_201"]}If you lack labels, sample production logs and manually annotate the top-10 results for a subset. Spend an hour on this; it pays back immediately. This file is your regression suite for every change below. Version it in git alongside the index build script.
Size and diversity
Include negative-style queries (things your docs don’t cover) to ensure you aren’t accidentally retrieving noise. A set skewed only to easy positives hides real failures.
Step 2: Compute baseline retrieval metrics
Pick your vector store and run each eval query. Compute recall@k and mean reciprocal rank (MRR). For Chroma:
import chromadb
from sklearn.metrics import recall_score
client = chromadb.PersistentClient(path="./chroma")
coll = client.get_collection("docs")
def recall_at_k(retrieved, relevant, k):
return len(set(retrieved[:k]) & set(relevant)) / len(relevant)
eval_data = [{"query": "..."}] # loaded from jsonl
recalls, mrrs = [], []
for item in eval_data:
res = coll.query(query_texts=[item["query"]], n_results=10)
ids = res["ids"][0]
recalls.append(recall_at_k(ids, item["relevant_ids"], 5))
for rank, i in enumerate(ids, 1):
if i in item["relevant_ids"]:
mrrs.append(1/rank)
break
else:
mrrs.append(0)
print("Recall@5:", sum(recalls)/len(recalls), "MRR:", sum(mrrs)/len(mrrs))A recall@5 below 0.7 signals the retriever is dropping context before the model ever sees it. That is the number to move. MRR tells you if the right doc surfaces at rank 1 or 3.
What to log
Record the query, returned IDs, and cosine scores to a CSV. You will need this trace in later steps to spot patterns (e.g., all misses are long queries). Recall@k is a retrieval-only metric; it does not care if the LLM later ignores the passage.
Step 3: Profile embedding space separation
Low recall often means the embedding model places relevant and irrelevant passages too close. Sample positive pairs (query, relevant doc) and negative pairs (query, random doc). Compute cosine similarity distributions.
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2")
def cos(a, b):
return np.dot(a, b) / (np.linalg.norm(a)*np.linalg.norm(b))
pos_sims, neg_sims = [], []
for item in eval_data:
q = model.encode(item["query"])
for rid in item["relevant_ids"]:
doc = get_doc(rid)
pos_sims.append(cos(q, model.encode(doc)))
for rid in random_ids(5):
neg_sims.append(cos(q, model.encode(get_doc(rid))))
print("Pos mean:", np.mean(pos_sims), "Neg mean:", np.mean(neg_sims))If the gap between positive and negative means is under 0.2, the embedding space is too flat. That points to model or chunking issues, not the vector DB. If you see positive mean 0.6 and negative 0.5, your retriever is essentially guessing. Plot histograms if you can; a visible overlap explains why threshold filters fail.
Step 4: Audit your chunking strategy
Retrieval quality lives or dies on chunk boundaries. This is core to debugging retrieval quality RAG. Common mistakes: splitting mid-sentence, chunks too large (noise dilutes signal), or no overlap (context fractured).
A sane starting point with token awareness:
def chunk_text(text, size=512, overlap=64):
tokens = text.split() # replace with tokenizer for precision
chunks = []
start = 0
while start < len(tokens):
chunk = " ".join(tokens[start:start+size])
chunks.append(chunk)
start += size - overlap
return chunksTest size variants (256, 512, 1024). Re-embed and re-run Step 2. If recall jumps at a smaller size, your original chunks buried the answer in filler. If larger size helps, the answer needed surrounding context.
Use a sentence tokenizer to avoid mid-sentence cuts:
import nltk
nltk.download('punkt')
from nltk.tokenize import sent_tokenize
def chunk_by_sentences(text, max_sents=10, overlap=2):
sents = sent_tokenize(text)
chunks = []
for i in range(0, len(sents), max_sents - overlap):
chunks.append(" ".join(sents[i:i+max_sents]))
return chunksThis respects natural boundaries and often beats fixed token windows.
Metadata filtering
If you have metadata (e.g., product area), check whether a filter clause is erroneously excluding relevant docs. Disable filters in a baseline run to isolate this. A missing where clause can silently cut recall in half.
Step 5: Swap or fine-tune the embedding model
If Step 3 showed poor separation, try a stronger model. Swap all-MiniLM-L6-v2 for multi-qa-mpnet-base-dot-v1 or a domain-specific one.
model = SentenceTransformer("multi-qa-mpnet-base-dot-v1")
# re-embed corpus and rebuild index, then re-run evalFor specialized vocabularies (legal, medical), fine-tune with SentenceTransformer.fit on positive pairs from your eval set. Even 200 pairs can shift the similarity gap noticeably. Weigh latency: a 400ms embed time may be fine for offline indexing but deadly for online re-embedding.
Step 6: Introduce a reranking stage
Dense retrieval gets candidates; a cross-encoder reranker reorders them by true relevance. This is the highest-leverage fix for debugging retrieval quality RAG after chunking is stable.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query, candidates):
pairs = [(query, c) for c in candidates]
scores = reranker.predict(pairs)
return [c for _, c in sorted(zip(scores, candidates), reverse=True)]
# in retrieval loop:
base = coll.query(query_texts=[q], n_results=20)["documents"][0]
ranked = rerank(q, base)Measure recall@5 again. Typical lift is 10–30% relative on noisy corpora, but verify on your own numbers. Reranking adds latency; cache scores for repeated queries.
Step 7: Validate end-to-end with a generation check
Retrieval metrics alone don’t guarantee good answers. Take the top-3 passages, feed them to a model, and judge faithfulness. If you route the generation call through n4n.ai’s OpenAI-compatible endpoint, you get per-token metering and automatic fallback across 240+ models without changing your client code.
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Answer only from context."},
{"role": "user", "content": f"Context: {ranked[:3]}\nQ: {q}"}
]
)Have a second LLM judge whether the answer is grounded. Use a strict prompt: “Is every claim backed by the context? Reply yes/no.” Track faithfulness score alongside recall.
How to verify success
You have fixed retrieval when:
- Recall@5 on the eval set rises above 0.85 (or improves >15% relative from baseline).
- Positive/negative cosine gap widens by at least 0.1 after model or chunking change.
- Reranker pushes relevant docs into top-3 for >90% of queries.
- End-to-end faithfulness judge scores ≥0.9.
Keep the eval set in CI. Every embedding or chunking PR should print these metrics. Debugging retrieval quality RAG is not a one-off task; it is a continuous guardrail that prevents silent regression as your corpus grows.