Debugging LlamaIndex retrieval pipelines step by step

Retrieval failures in RAG systems are silent: the LLM happily answers from the wrong context. Effective debugging LlamaIndex retrieval pipelines requires visibility into every node fetched before generation. This tutorial walks through instrumenting a pipeline, inspecting retrieved nodes, and fixing the most common retrieval defects with runnable code.

Prerequisites

• Python 3.10 or newer
• llama_index installed (pip install llama_index>=0.10.0)
• An OpenAI API key, or any OpenAI-compatible endpoint
• A minimal corpus you control (we’ll use inline Document objects, no file I/O needed)

pip install llama_index
export OPENAI_API_KEY="sk-..."

You should be comfortable with Python and basic RAG concepts. We will not cover deployment.

Build a minimal pipeline

Start with three documents and a vector index. This is the smallest useful LlamaIndex retrieval pipeline.

from llama_index.core import Document, VectorStoreIndex, Settings
from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-4o-mini")

docs = [
    Document(text="LlamaIndex is a data framework for LLM applications. It connects to 240+ models via gateways."),
    Document(text="Retrieval augmented generation grounds LLM answers in private documents."),
    Document(text="Debugging RAG requires inspecting the chunks fed to the prompt."),
]

index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine()

response = query_engine.query("How many models does LlamaIndex connect to?")
print(str(response))

Expected output (content may vary slightly by model):

LlamaIndex connects to 240+ models via gateways.

If you see a confident but wrong answer, the bug is almost always in retrieval, not the LLM.

Enable debug tracing

The fastest way to see what the retriever actually fetched is LlamaDebugHandler. Attach it to the global CallbackManager before running the query.

from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler

debug_handler = LlamaDebugHandler()
Settings.callback_manager = CallbackManager([debug_handler])

# Re-create the query engine so it picks up the new callback manager
query_engine = index.as_query_engine()
response = query_engine.query("How many models does LlamaIndex connect to?")

events = debug_handler.get_events()
for e in events:
    print(e.event_type, "-", e.payload.get("kwargs", {}).get("query_str", ""))

Expected output:

retrieve - How many models does LlamaIndex connect to?
llm - 

The retrieve event confirms the query string reached the retriever. If you don’t see it, the query engine was built before the callback manager was set.

Inspect retrieved nodes directly

Query engines hide the retriever. Call it explicitly to debug LlamaIndex retrieval pipelines at the node level.

retriever = index.as_retriever(similarity_top_k=2)
nodes = retriever.retrieve("How many models does LlamaIndex connect to?")

for i, n in enumerate(nodes):
    print(f"[{i}] score={n.score:.3f}")
    print(n.node.get_content()[:120])
    print("metadata:", n.node.metadata)
    print("---")

Expected output:

[0] score=0.842
LlamaIndex is a data framework for LLM applications. It connects to 240+ models via gateways.
metadata: {}
---
[1] score=0.613
Debugging RAG requires inspecting the chunks fed to the prompt.
metadata: {}
---

The first node is on-topic. The second is noise. If the top score is below ~0.7 on a small corpus, your embedding or chunking is suspect.

Common retrieval defects

When debugging LlamaIndex retrieval pipelines, these are the failures I see most:

1. Empty result list – similarity_top_k too high for a tiny index, or embedding dimension mismatch.
2. Low scores across the board – chunk size too large, or the query embedding is diluted by stopwords.
3. Right score, wrong node – metadata filters missing, or the vector store returning approximate neighbors.
4. Topic drift – no reranking, so the LLM sees the second node more than the first.

Print the scores and content. Do not trust the final answer as a signal.

Tune chunk size and re-index

Default chunk size is 1024 tokens, which is wrong for three short sentences. Reduce it and re-embed.

from llama_index.core.node_parser import SentenceSplitter

Settings.node_parser = SentenceSplitter(chunk_size=64, chunk_overlap=8)
index = VectorStoreIndex.from_documents(docs)
retriever = index.as_retriever(similarity_top_k=1)
nodes = retriever.retrieve("How many models does LlamaIndex connect to?")

print("top score:", nodes[0].score)
print(nodes[0].node.get_content())

Expected output:

top score: 0.891
LlamaIndex is a data framework for LLM applications. It connects to 240+ models via gateways.

Smaller chunks sharpen the cosine similarity. The noise node drops out entirely because top_k=1.

Add a deterministic evaluation check

You do not need an LLM to verify retrieval. Write a unit-test-style assertion.

def retrieval_contains(query: str, needle: str) -> bool:
    nodes = retriever.retrieve(query)
    return any(needle in n.node.get_content() for n in nodes)

assert retrieval_contains("How many models does LlamaIndex connect to?", "240+ models")
print("retrieval OK")

This fails loudly when someone changes the embedding model or corpus without updating the test.

Swap in a stable model endpoint

When debugging LlamaIndex retrieval pipelines, flaky model endpoints waste cycles. Point Settings.llm at an OpenAI-compatible gateway like n4n.ai to get automatic fallback when a provider is rate-limited or degraded, without touching the retrieval code.

Settings.llm = OpenAI(
    api_base="https://api.n4n.ai/v1",
    api_key="YOUR_N4N_KEY",
    model="gpt-4o-mini",
)

The rest of the pipeline—retriever, nodes, callback manager—stays identical. You only changed where the completion request lands.

Use a custom callback for structured logs

For CI debugging, emit JSON instead of printing to stdout.

import json
from llama_index.core.callbacks import BaseCallbackHandler

class JsonDebugHandler(BaseCallbackHandler):
    def __init__(self):
        self.events = []
    def on_event(self, event_type, payload, event_id, parent_id):
        if event_type == "retrieve":
            self.events.append({"event": event_type, "query": payload["kwargs"]["query_str"]})

handler = JsonDebugHandler()
Settings.callback_manager = CallbackManager([handler])
query_engine = index.as_query_engine()
query_engine.query("How many models does LlamaIndex connect to?")
print(json.dumps(handler.events))

Expected output:

[{"event": "retrieve", "query": "How many models does LlamaIndex connect to?"}]

Pipe this to a file and diff it across runs to catch regressions in query routing.

Recap

Debugging LlamaIndex retrieval pipelines is mostly about making the invisible visible: attach a callback, call the retriever directly, print scores, and assert on content. Tune chunk size before you tune prompts. Only after retrieval is correct should you spend a token on generation.