Common failure points in RAG pipeline design

Most RAG pipelines work great in demos and fall apart in production. The failure points aren’t mysterious — they’re the same handful of design decisions that look reasonable in isolation but compound under real traffic, real documents, and real user expectations. If you’ve shipped a RAG system that passed evals but users still complain “it doesn’t know anything,” this is probably why.

Chunking strategy determines your ceiling

The most common failure point happens before retrieval even runs. Engineers pick a chunk size (512 tokens, 1000 characters, “whatever LangChain defaults to”) and a splitter (recursive character, markdown, semantic) without validating that the resulting chunks actually contain answerable units of information.

Too large: You stuff irrelevant context into the generator, increasing latency, cost, and hallucination surface area. The model has to “find the needle” in a haystack you created.

Too small: You fracture concepts across chunks. A single procedure spanning three paragraphs becomes three orphaned fragments, none of which answer the question alone.

Wrong boundaries: Splitting mid-table, mid-code-block, or mid-argument destroys the semantic coherence that embeddings rely on.

# Naive approach — fails on structured docs
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", " ", ""]
)
chunks = splitter.split_text(raw_pdf_text)

# Better: structure-aware splitting with metadata preservation
from langchain.text_splitter import MarkdownHeaderTextSplitter
from langchain.document_loaders import UnstructuredPDFLoader

loader = UnstructuredPDFLoader("spec.pdf", mode="elements")
elements = loader.load()

# Preserve header hierarchy as metadata
header_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#", "h1"), ("##", "h2"), ("###", "h3")
    ]
)
chunks = []
for el in elements:
    if el.metadata.get("category") == "Table":
        # Keep tables intact with caption context
        chunks.append(el)
    else:
        md_chunks = header_splitter.split_text(el.page_content)
        for chunk in md_chunks:
            chunk.metadata.update(el.metadata)  # preserve source, page, headers
        chunks.extend(md_chunks)

The rule: chunk to the granularity of the questions you expect. If users ask “what’s the return policy for international orders?”, your chunks must contain complete policy sections, not scattered sentences. Validate by sampling 50 real queries and checking whether the ground-truth answer exists wholly within a single retrieved chunk.

Embedding model mismatch with retrieval task

Teams default to text-embedding-3-small or text-embedding-ada-002 because they’re cheap and fast. Then they wonder why retrieval fails on:

• Domain-specific terminology (legal, medical, financial)
• Code and technical documentation
• Non-English languages
• Negation and temporal reasoning (“show me policies before 2023”)

General-purpose embeddings optimize for semantic similarity in a broad sense. Your retrieval task is usually topical relevance + factual precision. These diverge.

# Evaluate embedding models on YOUR data, not MTEB
from sentence_transformers import SentenceTransformer
import numpy as np

candidates = [
    "text-embedding-3-small",
    "text-embedding-3-large", 
    "BAAI/bge-large-en-v1.5",
    "intfloat/e5-large-v2",
    "mixedbread-ai/mxbai-embed-large-v1",
]

# Build a labeled eval set: (query, relevant_chunk_ids)
eval_set = load_your_labeled_pairs("eval/retrieval_golden_set.jsonl")

def recall_at_k(model_name, k=10):
    model = SentenceTransformer(model_name)
    corpus_embeddings = model.encode(corpus_texts, batch_size=64, show_progress_bar=True)
    query_embeddings = model.encode([q for q, _ in eval_set], batch_size=64)
    
    hits = 0
    for i, (_, relevant_ids) in enumerate(eval_set):
        scores = cosine_similarity([query_embeddings[i]], corpus_embeddings)[0]
        top_k = np.argsort(scores)[-k:][::-1]
        if any(idx in relevant_ids for idx in top_k):
            hits += 1
    return hits / len(eval_set)

for m in candidates:
    print(f"{m}: recall@10 = {recall_at_k(m):.3f}")

Run this. The winner is often not the biggest model. bge-large-en-v1.5 frequently beats text-embedding-3-large on technical docs at 1/3 the dimension (1024 vs 3072), meaning faster ANN search and lower storage. But test on your corpus.

Tradeoff: Larger embeddings improve recall but increase vector DB memory, index build time, and query latency. If you’re serving 100 QPS, a 3072-dim index may need 3x the RAM of 1024-dim. Quantize (int8, binary) if you must use large dims — but validate recall doesn’t collapse.

Retrieval without reranking is gambling

Vector search returns semantically similar chunks. The user needs factually relevant chunks. These are different. A chunk discussing “API rate limits” is semantically similar to a query about “API rate limits for enterprise tier” but may not contain the enterprise-specific numbers.

Single-stage retrieval (vector only) typically achieves 60-75% recall@10 on real workloads. Adding a cross-encoder reranker pushes this to 85-95% — but at latency cost.

# Two-stage retrieval: vector ANN -> cross-encoder rerank
from sentence_transformers import CrossEncoder
import asyncio

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512)

async def retrieve_with_rerank(query: str, k: int = 5, candidate_k: int = 50):
    # Stage 1: fast vector search (candidate_k >> k)
    candidates = await vector_store.asimilarity_search(query, k=candidate_k)
    
    # Stage 2: precise reranking
    pairs = [(query, doc.page_content) for doc in candidates]
    scores = await asyncio.to_thread(reranker.predict, pairs)
    
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in ranked[:k]]

# Latency budget: vector search ~10-30ms, rerank 50 candidates ~50-100ms
# Total added latency: ~60-130ms. Worth it for most user-facing apps.

When to skip reranking: High-throughput internal tools where latency > precision, or when your vector search already achieves >90% recall@k on eval (rare). When to add a third stage (keyword/BM25): Hybrid search catches exact-match queries (error codes, model numbers, proper nouns) that dense embeddings miss.

# Hybrid: combine vector + BM25 scores
from rank_bm25 import BM25Okapi

bm25 = BM25Okapi([doc.split() for doc in corpus_texts])

def hybrid_search(query: str, k: int = 10, alpha: float = 0.5):
    vector_scores = vector_store.similarity_search_with_score(query, k=k*3)
    bm25_scores = bm25.get_scores(query.split())
    
    # Normalize and combine
    vec_norm = normalize([s for _, s in vector_scores])
    bm25_norm = normalize(bm25_scores)
    
    combined = alpha * vec_norm + (1 - alpha) * bm25_norm
    top_indices = np.argsort(combined)[-k:][::-1]
    return [corpus_docs[i] for i in top_indices]

Tune alpha on your eval set. Typical sweet spot: 0.3-0.7 depending on query type distribution.

Context window management: the silent killer

You retrieved 10 chunks. You stuff them all into the prompt. The model hallucinates because:

1. Irrelevant chunks distract — the model attends to noise
2. Contradictory chunks confuse — stale vs updated policies, conflicting specs
3. Token budget exhausted — no room for reasoning or citations

# Naive: dump everything
prompt = f"Context:\n{''.join(chunks)}\n\nQuestion: {query}\nAnswer:"

# Better: compress, deduplicate, cite
def build_context(chunks: List[Document], max_tokens: int = 4000) -> str:
    # 1. Deduplicate by content hash (common in crawled docs)
    seen = set()
    unique = []
    for c in chunks:
        h = hashlib.md5(c.page_content.encode()).hexdigest()[:16]
        if h not in seen:
            seen.add(h)
            unique.append(c)
    
    # 2. Score by reranker score + recency + source authority
    for c in unique:
        c.metadata["composite_score"] = (
            0.5 * c.metadata.get("rerank_score", 0) +
            0.3 * recency_score(c.metadata.get("date")) +
            0.2 * source_authority(c.metadata.get("source"))
        )
    unique.sort(key=lambda x: x.metadata["composite_score"], reverse=True)
    
    # 3. Pack with citations, respect token budget
    encoder = tiktoken.encoding_for_model("gpt-4o")
    context_parts = []
    token_count = 0
    
    for i, chunk in enumerate(unique):
        cited = f"[Doc {i+1}] {chunk.page_content}"
        cited_tokens = len(encoder.encode(cited))
        if token_count + cited_tokens > max_tokens:
            break
        context_parts.append(cited)
        token_count += cited_tokens
    
    return "\n\n".join(context_parts)

Critical: Reserve tokens for the answer, not just context. If your max context is 8k and you use 7.5k for retrieval, the model has 500 tokens to reason and cite. It will truncate mid-sentence or skip citations. Budget: max_context_tokens = model_context_window - expected_output_tokens - prompt_overhead - safety_margin.

Generation failures: citations, hallucination, refusal

Even with perfect retrieval, generation fails in predictable ways:

Missing citations: The model answers correctly but doesn’t reference sources. Users (and compliance) can’t verify.

Hallucinated citations: The model invents [Doc 3] references that don’t exist in context.

Over-refusal: “I don’t have enough information” when the answer is clearly in context — usually because the prompt doesn’t explicitly permit synthesis across chunks.

# Prompt template that reduces all three failure modes
RAG_PROMPT = """You are a precise technical assistant. Answer the user's question using ONLY the provided context.

Rules:
1. Cite every factual claim using [Doc N] format matching the context labels.
2. If multiple docs support a claim, cite all: [Doc 1][Doc 3].
3. If the context lacks the answer, say "The provided context does not contain this information."
4. Do not use external knowledge. Do not speculate.
5. Synthesize across docs when they collectively answer the question.

Context:
{context}

Question: {question}

Answer:"""

# Post-generation citation validation
def validate_citations(answer: str, context_docs: List[Document]) -> Tuple[bool, List[str]]:
    """Check that all [Doc N] references exist in context."""
    import re
    citations = re.findall(r'\[Doc (\d+)\]', answer)
    max_doc = len(context_docs)
    invalid = [c for c in citations if not (1 <= int(c) <= max_doc)]
    return len(invalid) == 0, invalid

Tradeoff: Strict citation prompts increase latency (more tokens) and can make answers verbose. For user-facing chat, accept some citation imperfection. For legal/medical/financial, enforce validation and regenerate on failure.

Evaluation blind spots

Teams evaluate retrieval (recall@k) and generation (BLEU/ROUGE vs reference answers) separately. This misses the compound failure: retrieval returns a relevant chunk but the generator ignores it, or retrieval misses but the generator hallucinates a plausible answer that passes human eval.

You need end-to-end eval on realistic queries with these dimensions:

# Minimal end-to-end eval framework
from dataclasses import dataclass
from typing import List, Optional
import jsonlines

@dataclass
class EvalCase:
    query: str
    expected_answer: str           # ground truth
    required_facts: List[str]      # atomic facts that must appear
    forbidden_claims: List[str]    # hallucinations to catch
    relevant_doc_ids: List[str]    # for retrieval eval

@dataclass
class EvalResult:
    query: str
    retrieved_ids: List[str]
    answer: str
    citations: List[int]
    
    # Computed metrics
    retrieval_recall: float
    fact_coverage: float           # % of required_facts present
    hallucination_rate: float      # % of forbidden_claims present
    citation_validity: bool
    citation_completeness: float   # % of factual sentences cited

def evaluate_case(case: EvalCase, result: EvalResult) -> EvalResult:
    # Retrieval
    retrieved_set = set(result.retrieved_ids)
    relevant_set = set(case.relevant_doc_ids)
    result.retrieval_recall = len(retrieved_set & relevant_set) / len(relevant_set) if relevant_set else 1.0
    
    # Fact coverage (simple substring check — upgrade to NLI for production)
    result.fact_coverage = sum(1 for f in case.required_facts if f.lower() in result.answer.lower()) / len(case.required_facts)
    
    # Hallucination check
    result.hallucination_rate = sum(1 for f in case.forbidden_claims if f.lower() in result.answer.lower()) / len(case.forbidden_claims) if case.forbidden_claims else 0.0
    
    # Citation validity
    result.citation_validity, _ = validate_citations(result.answer, [docs[i] for i in result.citations])
    
    return result

Build a golden set of 100-200 real queries with required facts, forbidden claims, and relevant doc IDs. Run this on every pipeline change. Track the composite score: 0.4 * retrieval_recall + 0.4 * fact_coverage + 0.1 * (1 - hallucination_rate) + 0.1 * citation_completeness.

What this catches that unit evals miss:

• Retriever finds doc but generator ignores it (low fact_coverage despite high retrieval_recall)
• Generator hallucinates plausible but forbidden claims (high hallucination_rate)
• Generator answers correctly but cites wrong docs (citation_validity = false)

Operational failure points: freshness, latency, cost

These aren’t “design” failures — they’re “didn’t think about operations” failures.

Stale data: Your vector index updates nightly. Legal pushes a policy change at 2 PM. Users get wrong answers for 22 hours. Solution: incremental upserts on document change events, not batch rebuilds.

# Event-driven index updates (simplified)
async def on_document_change(event: DocumentChangeEvent):
    if event.type == "delete":
        await vector_store.adelete(ids=[event.doc_id])
    else:
        chunks = chunk_document(event.content, event.metadata)
        embeddings = await embed_batch([c.page_content for c in chunks])
        await vector_store.aupsert(
            ids=[f"{event.doc_id}_{i}" for i in range(len(chunks))],
            vectors=embeddings,
            metadatas=[c.metadata for c in chunks],
            documents=[c.page_content for c in chunks]
        )

Tail latency: P95 latency is 800ms but P99 is 4s because one chunk triggers a slow cross-encoder call or the LLM hits a long generation. Solution: per-stage timeouts with graceful degradation.

async def retrieve_with_timeout(query: str, k: int = 5) -> List[Document]:
    try:
        # Vector search: hard 100ms timeout
        candidates = await asyncio.wait_for(
            vector_store.asimilarity_search(query, k=50),
            timeout=0.1
        )
    except asyncio.TimeoutError:
        log.warning("Vector search timeout, falling back to BM25")
        return bm25_fallback(query, k)
    
    try:
        # Rerank: hard 200ms timeout
        reranked = await asyncio.wait_for(
            rerank_candidates(query, candidates),
            timeout=0.2
        )
        return reranked[:k]
    except asyncio.TimeoutError:
        log.warning("Rerank timeout, returning vector results")
        return candidates[:k]

Cost explosion: You enable text-embedding-3-large + reranker + GPT-4o for every query. At 10k queries/day, embedding costs alone exceed $150/month. Solution: tiered routing — simple queries (faq, definitional) use small embeddings + no rerank + cheaper model; complex queries get the full pipeline.

async def route_query(query: str) -> PipelineConfig:
    # Cheap classifier (distilbert, ~5ms)
    complexity = await complexity_classifier.predict(query)
    
    if complexity == "simple":
        return PipelineConfig(
            embedding_model="text-embedding-3-small",
            use_reranker=False,
            generator_model="gpt-4o-mini",
            max_chunks=3
        )
    elif complexity == "complex":
        return PipelineConfig(
            embedding_model="text-embedding-3-large",
            use_reranker=True,
            generator_model="gpt-4o",
            max_chunks=8
        )
    else:  # "code" or "technical"
        return PipelineConfig(
            embedding_model="mixedbread-ai/mxbai-embed-large-v1",
            use_reranker=True,
            generator_model="gpt-4o",
            max_chunks=10
        )

The decisive takeaway

RAG pipeline failures are rarely about the LLM. They’re about data preparation that doesn’t match query patterns, retrieval optimized for the wrong similarity, context construction that ignores token economics, and evaluation that measures components instead of outcomes.

Fix order:

1. Build a golden eval set first — 100 real queries with required facts and relevant docs. You cannot improve what you don’t measure end-to-end.
2. Chunk for your queries, not your documents — validate that answers exist wholly in single chunks.
3. Add reranking — it’s the highest ROI retrieval improvement for user-facing apps.
4. Enforce citation discipline in the prompt — validate post-generation.
5. Route by query complexity — stop paying premium prices for simple lookups.

Everything else is optimization. The pipeline that passes your golden set at acceptable latency and cost is the one you ship.