LlamaIndex VectorStoreIndex query engine explained

This llamaindex vectorstoreindex query engine tutorial strips away the boilerplate and shows the exact call chain from raw documents to answered queries. You will see where the embeddings get computed, how retrieval parameters change answer quality, and which defaults will quietly burn tokens in production.

What the query engine actually abstracts

A VectorStoreIndex is not a database. It is a LlamaIndex structure that persists Node objects alongside their vector embeddings in whatever backing store you configured (in-memory, Chroma, Pinecone, etc.). The query engine built on top of it does two things: retrieve the k most similar nodes via approximate nearest neighbor search, then hand those nodes to an LLM to synthesize a final answer.

The abstraction is convenient, but it hides three cost centers: embedding computation at ingest, vector search at query time, and LLM token consumption during synthesis. If you treat it as a black box, you will overpay for both storage and inference.

Step 1: Load and chunk documents

Start with SimpleDirectoryReader. The default chunk size is 1024 characters with 200 overlap. That works for prose but is wrong for code or structured logs.

from llama_index.core import SimpleDirectoryReader

docs = SimpleDirectoryReader(
    input_dir="./data",
    recursive=True,
).load_data()

print(len(docs))  # raw files, not chunks yet

Chunking happens inside VectorStoreIndex.from_documents using Settings.chunk_size and Settings.chunk_overlap. Set them explicitly:

from llama_index.core import Settings

Settings.chunk_size = 512
Settings.chunk_overlap = 64

Pitfall: if you change chunk size after indexing, old nodes keep the old size. Re-index or use a fresh index.

Step 2: Configure embeddings and LLM

LlamaIndex uses global Settings for the embedding model and the LLM. Do not pass models per-call unless you have a reason.

from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)

If you want to avoid a hard dependency on one provider, point Settings.llm at an OpenAI-compatible endpoint such as n4n.ai, which fronts 240+ models and auto-fails over on rate limits while metering per token. The LlamaIndex client code does not change.

Settings.llm = OpenAI(
    api_base="https://api.n4n.ai/v1",
    api_key="your-key",
    model="anthropic/claude-3.5-sonnet",
)

Tradeoff: embedding models are dimension-sensitive. Switching from text-embedding-3-small (1536 dim) to a local BAAI/bge-small (384 dim) without rebuilding the index throws a shape mismatch at query time.

Step 3: Build the index

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(
    docs,
    show_progress=True,
)

This computes embeddings for every chunk synchronously. For large corpora, use VectorStoreIndex.from_documents with a transformations pipeline or ingest via IngestionPipeline to parallelize and cache.

Persist if you need reuse:

index.storage_context.persist(persist_dir="./storage")

Step 4: Construct the query engine

The as_query_engine method is where retrieval and synthesis meet. The two parameters that matter most are similarity_top_k and response_mode.

query_engine = index.as_query_engine(
    similarity_top_k=4,
    response_mode="compact",
    streaming=False,
)

• similarity_top_k: number of nodes retrieved. Default is 2 in older versions, 4 in recent ones. More nodes = more context but higher LLM cost and potential noise.
• response_mode: "compact" concatenates retrieved text into the prompt (fast, cheap). "tree_summarize" does a hierarchical summarize (expensive, better for broad questions). "no_text" returns only retrieved nodes, skipping the LLM call entirely.

For a llamaindex vectorstoreindex query engine tutorial, the minimal run looks like:

response = query_engine.query("What is the refund policy for annual plans?")
print(response.response)
for node in response.source_nodes:
    print(node.score, node.text[:80])

Step 5: Add metadata filters

Raw vector search ignores document structure. If your corpus mixes invoices and manuals, filter at retrieval:

from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

filters = MetadataFilters(
    filters=[MetadataFilter(key="doc_type", value="manual")]
)

query_engine = index.as_query_engine(
    similarity_top_k=4,
    filters=filters,
)

This pushes the filter to the vector store before ANN search. Not all backends support it equally; in-memory and Chroma do, some hosted ones partial.

Common pitfalls and tradeoffs

Chunk size too large. A 1024-char chunk may span two topics. The retriever returns the whole chunk, polluting context. Drop to 256–512 for dense technical docs.

Top-k too high. Setting similarity_top_k=10 feels safer but routinely pushes prompt size past 8k tokens for marginal recall gains. Measure answer faithfulness at k=3,4,5 before scaling.

Reranking after retrieval. If precision matters, add a cross-encoder rerank as a node postprocessor:

from llama_index.core.postprocessor import SentenceTransformerRerank

rerank = SentenceTransformerRerank(
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
    top_n=3,
)
query_engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[rerank],
)

This retrieves 10, reranks, keeps 3. Cost: one extra model load and CPU/GPU inference per query.

Streaming vs latency. streaming=True improves perceived latency but breaks response.response (it becomes a generator). Use response.print_response_stream() or iterate response.response_gen.

No cache for embeddings. Re-indexing the same files recomputes embeddings. Use IngestionPipeline with a Docstore and VectorStore that support dedupe, or cache at the embedder level.

Production hardening checklist

1. Pin Settings.chunk_size and Settings.embed_model in version control.
2. Use response_mode="compact" unless evaluation shows otherwise.
3. Expose similarity_top_k as a tunable env var; default 4.
4. Add metadata filters at ingest so you can scope queries per tenant.
5. Wrap the LLM endpoint in a client that honors fallback. In this llamaindex vectorstoreindex query engine tutorial we showed one gateway approach; the key is that LlamaIndex does not care who answers as long as the API is OpenAI-shaped.

When not to use VectorStoreIndex

If your data is tabular or requires exact lookup, a SQLIndex or KeywordTableIndex will beat vector search. VectorStoreIndex shines when semantic similarity is the query intent. For hybrid, compose a RetrieverQueryEngine with both vector and BM25 retrievers and a union postprocessor.

The query engine is a thin, configurable layer. Own the chunking, the retrieval count, and the synthesis mode, and the default LlamaIndex stack will carry a real RAG feature into production without surprises.