LlamaIndex vector store index debugging starts with one uncomfortable truth: most retrieval failures are not model problems, they are index mismatches. You built the index with one embedding model, then queried it with another, or you changed chunk size and forgot to rebuild. This guide gives an ordered path to find and fix those mismatches before they reach production.
1. Confirm the embedding model is identical
The fastest way to break retrieval is to embed documents and queries with different models. LlamaIndex does not magically align dimensions or semantic spaces. If your index was built with text-embedding-3-small and you query with text-embedding-ada-002, cosine distances will be meaningless.
Print the embedding model and its dimension from both sides:
from llama_index.embeddings.openai import OpenAIEmbedding
build_emb = OpenAIEmbedding(model="text-embedding-3-small")
query_emb = OpenAIEmbedding(model="text-embedding-3-small")
print(build_emb.model_name, build_emb.dim)
print(query_emb.model_name, query_emb.dim)When loading an existing index, explicitly pass the same embed_model. If you omit it, LlamaIndex falls back to a default that may differ across versions:
index = VectorStoreIndex.load_from_storage(storage, embed_model=build_emb)A subtle trap: provider-side model aliases. OpenAI has upgraded text-embedding-3-small silently in the past. Pin the exact version if your gateway supports it. If you front embedding requests with a gateway such as n4n.ai, its OpenAI-compatible endpoint addresses 240+ models and may automatically fall back to a different embedding model when a provider is degraded. Disable fallback for indexing jobs and pin the model id.
2. Verify document chunking parameters
Even with identical embeddings, different chunk sizes produce different vector representations. LlamaIndex’s SentenceSplitter defaults have changed between minor versions. If you indexed with chunk_size=512 and later queried with a default of 1024, your nodes will not match.
Inspect the node content and reconstruct the splitter:
from llama_index.core.node_parser import SentenceSplitter
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = splitter.get_nodes_from_documents(documents)
for n in nodes[:3]:
print(n.get_content(), n.metadata.get("chunk_size"))Store the chunking config in your index metadata. A simple pattern is to write a config.json next to the persisted index and assert on load:
import json, hashlib
config = {"chunk_size": 512, "chunk_overlap": 64, "embed_model": "text-embedding-3-small"}
config_hash = hashlib.md5(json.dumps(config, sort_keys=True).encode()).hexdigest()If the hash mismatches at query time, fail fast instead of serving silent garbage.
3. Inspect the vector store schema and metadata
LlamaIndex abstracts many vector databases, but each has its own metadata rules. Chroma coerces types; Pinecone requires string keys. A mismatch here means your filtered queries return empty results even though vectors exist.
For Chroma, check the collection directly:
import chromadb
client = chromadb.PersistentClient(path="./chroma")
col = client.get_collection("my_index")
print(col.count(), col.metadata)Pull a few records and verify the metadata keys match what your retriever filters on:
data = col.get(limit=5, include=["metadatas"])
print(data["metadatas"])Common pitfall: indexing with metadata={"source": 123} (int) but querying with {"source": "123"} (string). The filter silently fails. Normalize all metadata to strings at ingest.
4. Check index persistence and loading code
LlamaIndex vector store index debugging often reveals that the index was never actually loaded from the expected location. StorageContext.persist() writes to disk, but VectorStoreIndex.load_from_storage requires the same vector store type.
If you use an external vector store (Chroma, Pinecone), persisting the LlamaIndex docstore is not enough—the vectors live in the DB. Your load code should look like:
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
chroma_client = chromadb.PersistentClient(path="./chroma")
chroma_col = chroma_client.get_or_create_collection("my_index")
vector_store = ChromaVectorStore(chroma_collection=chroma_col)
storage = StorageContext.from_defaults(vector_store=vector_store, persist_dir="./storage")
index = VectorStoreIndex.from_vector_store(vector_store, embed_model=build_emb)Do not call load_from_disk expecting it to rebuild the Chroma collection. It only loads the docstore and index struct.
5. Reproduce with a minimal query harness
Once the config is verified, isolate the retrieval call. A minimal harness removes application noise:
retriever = index.as_retriever(similarity_top_k=3)
nodes = retriever.retrieve("What is the refund policy?")
for n in nodes:
print(n.score, n.node_id, n.get_content()[:80])Compare this against a direct vector store query using the same embedding. If the scores differ wildly, your LlamaIndex retriever is adding a transform (e.g., reranking) you forgot about.
For Chroma directly:
query_emb = query_emb.get_query_embedding("What is the refund policy?")
res = col.query(query_embeddings=[query_emb], n_results=3)
print(res["distances"], res["ids"])If the direct query returns sensible hits but LlamaIndex does not, the bug is in the LlamaIndex query pipeline, not the index.
6. Validate distance metrics and top-k sanity
LlamaIndex assumes cosine similarity for many stores, but Chroma defaults to L2, Pinecone to cosine. A metric mismatch reorders results even with correct embeddings.
Set the metric explicitly when creating the collection:
chroma_col = chroma_client.create_collection("my_index", metadata={"hnsw:space": "cosine"})Then sanity-check top-k: embed a sentence that appears verbatim in a source document. Its nearest neighbor should be that document with score near 1.0 (cosine). If the top score is 0.2, something is off—likely a different embed model or normalized text.
Tradeoff: switching metrics requires reindexing. Do it once, in a script, and assert the metric in your config hash.
7. Handle provider routing and caching
When embeddings are served through a proxy, cache-control hints and fallback routing can silently alter outputs. A gateway that honors client routing directives but falls back on rate limits will produce vectors from a different model mid-build. Your index then contains a mixture of spaces.
Mitigation: set cache_control headers if your embed client supports them, and lock the route for batch jobs. For ad-hoc debugging, run a local embedding model to eliminate the network variable entirely.
Common pitfalls and tradeoffs
- Version drift: LlamaIndex minor versions change defaults. Pin the version in your requirements and store it in the config hash.
- Metadata typing: Always stringify metadata values at ingest.
- Rebuilding cost: Reindexing large corpora is expensive. Use the config hash to avoid unnecessary rebuilds, but accept that a mismatch forces a full rebuild.
- Hybrid search: Adding BM25 alongside vectors complicates debugging. Disable hybrid mode until the dense path is verified.
LlamaIndex vector store index debugging is mostly disciplined configuration management. Embed the same way, chunk the same way, store the same way, and assert it on every load. The retrieval bugs that survive that discipline are the interesting ones.