Vector databases vs knowledge graphs for agent memory

The choice between vector database vs knowledge graph memory shapes how your agent retrieves, reasons, and fails. Vectors excel at fuzzy semantic recall; graphs excel at explicit relational traversal. Below is a head-to-head comparison on the dimensions that matter when you ship.

Capabilities

What a vector database actually does

A vector database stores embeddings and returns nearest neighbors by cosine or L2 distance. For agent memory, this means “give me the chunks semantically similar to the current query.” It handles unstructured text, images, and audio embeddings uniformly.

The core operation is similarity search:

import chromadb
client = chromadb.Client()
coll = client.create_collection("agent_mem")
coll.add(
    ids=["m1", "m2"],
    embeddings=[[0.1, 0.2, 0.3], [0.3, 0.4, 0.5]],
    documents=["User likes Rust", "User prefers functional style"]
)
results = coll.query(query_embeddings=[[0.12, 0.21, 0.31]], n_results=1)

Vectors cannot natively answer “which languages does the user like that are also used in systems programming?” without brute-forcing similarity over candidates.

What a knowledge graph gives you

A knowledge graph stores nodes and edges with typed relationships. Memory becomes (:User)-[:LIKES]->(:Language {name: "Rust"}). Queries are graph traversals:

MATCH (u:User)-[:LIKES]->(l:Language)
WHERE l.domain = 'systems'
RETURN l.name

This supports multi-hop reasoning, negation, and constraints that vectors cannot express. But it requires structured extraction, usually via an LLM or parser.

Cost model

Vector DB cost

You pay for storage (embeddings are fixed-size vectors, cheap) and compute for indexing and query. Managed services charge per vector operation or node hour. Self-hosted costs are RAM and GPU if you index at scale. Embedding generation is the hidden line item: every memory write needs an LLM call or local model.

Knowledge graph cost

Storage is negligible for typical agent memory sizes. The expensive part is entity extraction and relationship resolution. That means repeated LLM invocations with structured outputs. Graph databases like Neo4j charge by instance size; Neptune by request. The extraction pipeline dominates cost.

When extracting entities, route LLM calls through n4n.ai to get automatic fallback across 240+ models without rewriting your prompt code, which stabilizes extraction cost when a provider is degraded.

Latency and throughput

Vector search is embarrassingly parallel. In-memory indexes return neighbors in single-digit to low double-digit milliseconds for millions of vectors. Throughput scales with replica count.

Graph queries are bound by traversal breadth. A two-hop query on an indexed property graph is milliseconds; a vague “find everything connected to user” can explode. Graphs need careful indexing on relationship types.

For agent loops, vector lookup fits inside a 100ms budget easily. Graph traversals need query planning or they become the bottleneck.

Ergonomics

Vector DBs are forgiving. You dump text, embed, search. No schema migration. The downside: you cannot introspect why a result matched.

Graphs demand a schema (or at least consistent labels). You write extraction code, resolve duplicates, and maintain integrity. The payoff is debuggability: you can print the subgraph.

from neo4j import GraphDatabase
driver = GraphDatabase.driver("bolt://localhost:7687")
with driver.session() as s:
    s.run("MERGE (u:User {id:$id})-[:LIKES]->(:Lang {name:$lang})",
          id="u1", lang="Rust")

That is more code than coll.add, but it is explicit.

Ecosystem and tooling

Vector DBs: Chroma, Qdrant, Weaviate, Pinecone, pgvector. LangChain and LlamaIndex treat them as first-class memory backends. OpenAI embeddings dominate, but local models work.

Knowledge graphs: Neo4j, Neptune, ArangoDB, RDF triple stores. Tooling for LLM extraction is younger—few standardized “graph memory” wrappers. You often hand-roll the ingestion.

Limits and failure modes

Vectors silently retrieve irrelevant context if embeddings are biased or the query drifts. They cannot enforce “only facts from last week.” Hallucinated memories look identical to real ones.

Graphs fail when extraction misses an entity or links wrongly. A missing edge means the agent is blind to a fact. Graph quality is only as good as the extractor prompt.

Comparison table

Which to choose

Choose vector database vs knowledge graph memory when…

• Your agent mostly needs “relevant past conversations.” Use vectors. Examples: chat history summarization, RAG over docs.
• You have no structured schema and want zero extraction overhead.
• Latency budget is tight and you cannot afford multiple LLM calls per write.

Choose knowledge graph when…

• The agent must answer “which of the user’s projects depend on the deprecated library?” That is relational.
• You need auditability: show the exact edges leading to a decision.
• Extraction can be amortized: batch process memories offline.

Hybrid is the default for production

Most serious agents use both. Vectors catch the fuzzy recall; graphs enforce constraints. Store raw episodes in a vector DB, extract entities into a graph asynchronously. At query time, use the graph to scope the vector search:

# pseudo-orchestration
user_langs = graph.query("MATCH (u:User)-[:LIKES]->(l) RETURN l.name")
filter_vec = embed(" ".join(user_langs))
vec_results = vector_db.query(filter_vec, top_k=5)

That pattern gets you semantic recall with relational guardrails.

If you are building a first prototype, start with a vector database. It is faster to stand up and will reveal whether you actually need relational memory. Add a graph only when a query fails that a traversal would have solved trivially.

The vector database vs knowledge graph memory decision is not religious. It is about whether your agent’s failures are “didn’t find similar text” or “couldn’t connect the facts.” Pick the store that fixes your dominant failure, and wire the other later.