LlamaIndex knowledge graph vs vector index compared

If you’re building a RAG system with LlamaIndex, the index choice shapes everything downstream: query latency, token spend, maintenance burden, and the kinds of questions your system can actually answer. Vector indexes are the default for a reason — they’re fast, cheap, and work well for semantic similarity. Knowledge graphs unlock multi-hop reasoning and explicit relationship traversal, but they demand more upfront extraction work and query-time compute. This comparison breaks down the trade-offs across the dimensions that matter in production.

How they work

A vector index embeds document chunks into a high-dimensional space and retrieves by cosine similarity (or dot product, or whatever metric your vector store supports). The retrieval path is straightforward: embed the query, search the index, return top-k chunks, stuff them into the prompt. LlamaIndex’s VectorStoreIndex wraps this with a clean abstraction over 40+ vector stores — Pinecone, Weaviate, Qdrant, pgvector, you name it.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=5)
response = query_engine.query("What were the Q3 revenue drivers?")

A knowledge graph index takes a different approach. During ingestion, an LLM extracts entities and relationships from your documents, storing them as triples (subject, predicate, object) in a graph store — typically Neo4j, FalkorDB, or Kuzu. At query time, the system can traverse edges, follow relationship chains, and assemble context from disconnected but related nodes. LlamaIndex’s KnowledgeGraphIndex supports both LLM-based extraction and spaCy-based extraction for lower-cost pipelines.

from llama_index.core import KnowledgeGraphIndex
from llama_index.graph_stores.neo4j import Neo4jGraphStore

graph_store = Neo4jGraphStore(
    username="neo4j",
    password="password",
    url="bolt://localhost:7687",
    database="neo4j"
)
index = KnowledgeGraphIndex.from_documents(
    documents,
    graph_store=graph_store,
    max_triplets_per_chunk=10,
    include_embeddings=True  # hybrid retrieval
)

The include_embeddings=True flag is important — it creates a hybrid index where vector similarity narrows the candidate subgraph before graph traversal. This is the production pattern.

Capabilities: what each enables

Vector indexes excel at “find me content semantically similar to this query.” They handle paraphrase, synonyms, and fuzzy intent well. They fail at questions requiring explicit relationship chains: “Which suppliers does Company A share with Company B?” or “Trace the regulatory approval path from Drug X to Market Y.” These require joining across documents via named entities — exactly what knowledge graphs model.

Knowledge graphs also support structured queries. You can ask “Show me all COMPETES_WITH relationships in the tech sector” and get a precise subgraph back. This enables analytical workflows that vector search cannot: impact analysis, dependency mapping, lineage tracing. The trade-off is recall — if the extraction missed a relationship, it doesn’t exist in the graph. Vector search degrades gracefully; graph extraction fails silently.

Cost model: ingestion and query

Vector index ingestion cost scales linearly with document count and chunk size. One embedding call per chunk. At current text-embedding-3-small pricing ($0.02/1M tokens), a 10K document corpus at 512-token chunks costs roughly $1-2 in embedding calls. Storage is cheap — a few GB in any vector store.

Knowledge graph ingestion is an order of magnitude more expensive. Each chunk triggers an LLM call for entity/relationship extraction. With GPT-4o-mini at $0.15/1M input tokens, that same 10K document corpus costs $50-200 depending on max_triplets_per_chunk and retry logic. You also pay for graph database hosting — Neo4j Aura professional starts at $65/month. FalkorDB and Kuzu are lighter but less battle-tested at scale.

Query-time cost flips the script. Vector search is one embedding call + ANN lookup (~10-50ms). Knowledge graph queries often require multiple LLM rounds: entity extraction from the query, subgraph retrieval, possibly multi-hop traversal, then synthesis. A complex graph query can consume 5-10x the tokens of a vector query. If you’re serving high-volume traffic, this shows up in your inference bill fast.

Latency and throughput

Vector indexes are built for throughput. ANN search on HNSW or IVF indexes returns results in single-digit milliseconds at millions of vectors. LlamaIndex’s async query engine (aquery) lets you saturate your vector store’s connection pool. We’ve seen 200+ QPS on modest pgvector hardware with proper connection pooling.

Knowledge graph queries are latency-bound by traversal depth and LLM calls. A 2-hop Cypher query on Neo4j takes 5-20ms, but the surrounding LLM orchestration (query planning, entity linking, answer synthesis) pushes end-to-end latency to 2-5 seconds. Throughput is limited by your LLM rate limits, not the graph store. If you need sub-500ms p99 at scale, vector wins. If you can tolerate 3-5 seconds for richer answers, graph becomes viable.

Ergonomics: setup, debugging, maintenance

Vector indexes are boring in a good way. VectorStoreIndex.from_documents() works out of the box. Debugging retrieval means checking top-k chunks and similarity scores — visible, interpretable. Re-indexing is a one-liner. Schema changes don’t exist; you just re-embed.

Knowledge graphs demand schema discipline. You define allowed entity types and relationship types upfront (or let the LLM invent them, which creates chaos). The extraction prompt is a critical production surface — small changes cascade into graph topology changes. Debugging means inspecting Cypher queries, checking node degrees, verifying that “Apple” the company didn’t merge with “Apple” the fruit. Re-indexing requires clearing the graph store and re-running extraction — a multi-hour operation at scale.

LlamaIndex’s PropertyGraphIndex (the newer, more flexible API) improves this with explicit schema definition and custom extractors, but the fundamental complexity remains. You’re running a graph database. That means backups, indexes on relationship properties, memory configuration, and the occasional CALL db.index.fulltext.createNodeIndex when full-text search becomes necessary.

Ecosystem and integrations

Vector stores are commoditized. Every cloud has a managed offering. LlamaIndex supports all of them with consistent APIs. You can swap Pinecone for Qdrant for pgvector with a config change. Hybrid search (vector + BM25) is a one-line toggle in most stores. Rerankers (Cohere, Jina, BGE) plug in via NodePostprocessor.

Graph stores are less portable. Neo4j dominates production; FalkorDB and Kuzu are promising but younger. LlamaIndex’s Neo4jGraphStore is the most mature integration. Cypher query generation from natural language works well with GPT-4 class models but degrades with smaller models. There’s no standard “reranker for graph traversal” — you’re building custom post-processing if you need it.

Multi-tenancy is another gap. Vector stores handle namespaces or partitioned indexes natively. Graph stores typically require separate databases or careful label-based isolation. If you’re building a multi-tenant RAG product, vector indexes are significantly easier to operate.

Limits: scale, context, freshness

Vector indexes scale to billions of vectors with proper sharding. Context window limits are handled by similarity_top_k and response_mode (compact, tree_summarize, etc.). Freshness is trivial — upsert new vectors, delete old ones. Most vector stores support real-time updates.

Knowledge graphs hit practical limits around 10-50M nodes on a single Neo4j instance. Beyond that you need fabric (sharding) or a different architecture. Context window pressure is real — a retrieved subgraph with 200 nodes and their properties can exceed 128K tokens before synthesis. You need aggressive subgraph sampling or summarization. Freshness is painful: updating a relationship means finding the affected nodes, deleting old triples, inserting new ones, and maintaining referential integrity. There’s no “upsert triple” equivalent to vector upsert.

Comparison table

Which to choose

Choose vector index when:

• Your queries are “find relevant information about X” — fact lookup, summarization, semantic search
• You need sub-500ms latency at any meaningful QPS
• Your corpus changes frequently (daily or hourly ingest)
• You’re building a multi-tenant product with isolation requirements
• Your team has no graph database ops experience
• Budget is constrained — vector is 10-50x cheaper end-to-end

Choose knowledge graph when:

• Users ask “how does A relate to B?” or “trace the path from X to Y” — supply chain, regulatory, legal, biomedical
• You need to answer analytical questions over the corpus: “Which entities have the most INFLUENCES relationships?”
• The domain has stable, well-defined ontologies (financial regulations, biomedical ontologies, legal citations)
• You can tolerate 3-5s latency and lower throughput
• You have or can hire graph database expertise
• The extraction schema is stable enough to justify the upfront investment

Choose hybrid (PropertyGraphIndex with include_embeddings=True) when:

• You need both semantic breadth and relationship depth
• Your queries mix “find me docs about X” with “how does X connect to Y?”
• You can route query types to different retrieval paths (vector for broad recall, graph for specific traversals)
• You’re willing to pay the ingestion cost once and maintain two retrieval paths

The hybrid approach is where most production systems land eventually. Start with vector. Add graph when you have concrete query patterns that vector cannot satisfy — not before. The ingestion cost and operational complexity of graphs are real, and they don’t pay off until you’re actually using the relationships.