LlamaIndex vs LangChain: which scales past a million vectors

If you’re building a RAG system that needs to handle a million or more vectors, the framework choice stops being about developer experience and starts being about memory pressure, query latency, and whether your index rebuilds fit in a maintenance window. Both LlamaIndex and LangChain can technically ingest that volume, but they optimize for different failure modes. I’ve run both in production at this scale; here’s what actually matters.

Architecture and data model

LlamaIndex treats the index as a first-class object. You construct a VectorStoreIndex (or SummaryIndex, TreeIndex, KeywordTableIndex) over Document objects, and the framework handles chunking, embedding, and persistence through a pluggable VectorStore abstraction. The index knows its own topology — you can swap the backend from Pinecone to Weaviate to a local SimpleVectorStore without rewriting ingestion logic.

LangChain treats vector stores as one of many Retriever implementations. You instantiate a Chroma, FAISS, PGVector, or Milvus store directly, call add_documents, and get back a retriever that implements get_relevant_documents. The framework doesn’t own the index lifecycle; you do. This means more boilerplate but also more control over sharding, batch sizing, and custom metadata filters.

# LlamaIndex: index-centric
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.pinecone import PineconeVectorStore

vector_store = PineconeVectorStore(index_name="prod-docs")
index = VectorStoreIndex.from_documents(
    SimpleDirectoryReader("./data").load_data(),
    vector_store=vector_store,
    show_progress=True,
)
query_engine = index.as_query_engine(similarity_top_k=10)

# LangChain: store-centric
from langchain_community.vectorstores import Pinecone
from langchain_openai import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter

embeddings = OpenAIEmbeddings()
vectorstore = Pinecone.from_documents(
    documents=text_splitter.split_documents(docs),
    embedding=embeddings,
    index_name="prod-docs",
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

At million-vector scale, LlamaIndex’s VectorStoreIndex adds a lightweight coordination layer (docstore, index store, vector store) that can become a bottleneck if you’re not using a managed backend. LangChain pushes that complexity to the vector store itself — which is fine if your store handles it, painful if it doesn’t.

Ingestion throughput and memory profile

LlamaIndex’s ingestion pipeline (IngestionPipeline) supports parallel document processing with configurable num_workers and batch embedding calls. The SimpleVectorStore (in-memory) will OOM around 500k vectors on a 32 GB machine. For production, you need a persistent backend. The framework batches embeddings at 100 documents by default; tune embed_batch_size to match your provider’s rate limits.

LangChain’s VectorStore.add_documents is a thin wrapper. Throughput depends entirely on the underlying store’s batch API. FAISS.add is single-threaded and memory-hungry — expect 2–3 GB per million 1536-dim vectors plus index overhead. PGVector with COPY can sustain 5k–10k vectors/sec on modest hardware if you disable indexes during bulk load and rebuild after.

# LlamaIndex: tuned ingestion pipeline
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512, chunk_overlap=64),
        OpenAIEmbedding(embed_batch_size=256),
    ],
    vector_store=vector_store,
    docstore=docstore,
)
nodes = pipeline.run(documents=documents, num_workers=4, show_progress=True)

# LangChain: PGVector bulk load pattern
from langchain_postgres import PGVector
from sqlalchemy import create_engine, text

engine = create_engine("postgresql://...")
with engine.connect() as conn:
    conn.execute(text("ALTER TABLE langchain_pg_embedding DISABLE TRIGGER ALL;"))
    conn.execute(text("DROP INDEX IF EXISTS langchain_pg_embedding_embedding_idx;"))

vectorstore = PGVector(
    connection=engine,
    embeddings=embeddings,
    collection_name="prod_docs",
    use_jsonb=True,
)
vectorstore.add_documents(docs, batch_size=5000)

with engine.connect() as conn:
    conn.execute(text("CREATE INDEX ON langchain_pg_embedding USING hnsw (embedding vector_cosine_ops);"))
    conn.execute(text("ALTER TABLE langchain_pg_embedding ENABLE TRIGGER ALL;"))

Query latency and retrieval quality

Both frameworks delegate ANN search to the vector store. The difference is in pre- and post-processing.

LlamaIndex’s QueryEngine composes retrieval, reranking, and synthesis in a single call. You get similarity_top_k, vector_store_query_mode (default, hybrid, sparse), and built-in NodePostprocessor hooks for rerankers (Cohere, Jina, BGE), metadata filtering, and deduplication. The ResponseSynthesizer handles tree-summarize or compact modes for long-context answers.

LangChain’s Retriever returns documents. You compose reranking and generation yourself — typically via ContextualCompressionRetriever wrapping a CrossEncoderReranker, then a RetrievalQA or custom chain. More pieces to wire, but each is swappable.

# LlamaIndex: rerank + synthesize in one call
from llama_index.core.postprocessor import CohereRerank
from llama_index.core.response_synthesizers import TreeSummarize

query_engine = index.as_query_engine(
    similarity_top_k=50,
    node_postprocessors=[CohereRerank(top_n=10, model="rerank-v3.5")],
    response_synthesizer=TreeSummarize(),
)
response = query_engine.query("What are the Q3 revenue drivers?")

# LangChain: explicit composition
from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank
from langchain.chains import RetrievalQA

compressor = CohereRerank(model="rerank-v3.5", top_n=10)
compression_retriever = ContextualCompressionRetriever(
    base_retriever=retriever,
    base_compressor=compressor,
)
qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=compression_retriever,
    chain_type="stuff",
)
result = qa.invoke({"query": "What are the Q3 revenue drivers?"})

At million-vector scale, the reranker call dominates latency (100–300 ms for 50 candidates). LlamaIndex’s integrated pipeline avoids an extra network round-trip between retrieval and rerank if you’re using a managed index that supports server-side reranking (Pinecone, Weaviate). LangChain gives you the hooks to implement the same optimization but doesn’t provide it out of the box.

Metadata filtering and hybrid search

LlamaIndex’s MetadataFilters and FilterOperator map to the vector store’s native filter syntax. The VectorStoreIndex translates a MetadataFilter spec into Pinecone’s $eq/$in, Weaviate’s where, or Qdrant’s Filter at query time. You define filterable fields at index creation via metadata_fields on the VectorStore constructor.

LangChain relies on the vector store’s search_kwargs — you pass raw filter dicts. This is more flexible (you can use store-specific operators) but less portable. If you switch from Pinecone to Qdrant, you rewrite every filter.

# LlamaIndex: portable metadata filters
from llama_index.core.vector_stores import MetadataFilters, FilterCondition

filters = MetadataFilters(
    filters=[
        ("department", "eq", "engineering"),
        ("year", "in", [2023, 2024]),
    ],
    condition=FilterCondition.AND,
)
query_engine = index.as_query_engine(filters=filters)

# LangChain: store-specific filters
retriever = vectorstore.as_retriever(
    search_kwargs={
        "k": 10,
        "filter": {"department": {"$eq": "engineering"}, "year": {"$in": [2023, 2024]}},
    }
)

Hybrid search (dense + sparse) follows the same pattern. LlamaIndex’s VectorStoreQueryMode.HYBRID works with any backend that supports it. LangChain requires store-specific hybrid_search calls or a custom retriever.

Ecosystem and operational maturity

LlamaIndex’s VectorStoreIndex integrates with 40+ vector stores via a consistent interface. The StorageContext abstraction means you can persist the docstore/index store to Redis, MongoDB, or Postgres while the vector store lives elsewhere. This matters at scale: you want the docstore (document → node mapping) on fast KV, the vector store on ANN-optimized hardware, and the index store (graph structure) somewhere durable.

LangChain has broader LLM/tool/agent integrations but thinner vector store abstractions. The VectorStore base class is minimal; each integration reimplements batching, filtering, and error handling. Community-maintained stores vary in quality — some lack async support, some don’t implement max_marginal_relevance_search, some leak connections under load.

Both frameworks support async. LlamaIndex’s aquery and aretrieve are first-class. LangChain’s ainvoke on chains and aget_relevant_documents on retrievers work but require the underlying store to implement async (many don’t).

Cost model at scale

The framework itself is free. Your costs are:

• Embeddings: ~$0.13/M tokens (OpenAI text-embedding-3-small) or self-hosted (BGE, E5) on GPU. A million 512-token chunks ≈ 500M tokens ≈ $65 per full re-embed.
• Vector store: Pinecone serverless ~$0.50/GB/month + $0.005/1k queries. Weaviate Cloud ~$0.35/GB/month. Self-hosted Qdrant/Milvus/PGVector on EC2: ~$200–500/month for a 3-node cluster handling 1M vectors.
• Reranking: Cohere rerank-v3.5 ~$1/1k queries. At 10k queries/day with top-50 rerank → $300/month.
• LLM synthesis: Depends on context window and model. GPT-4o-mini with 8k context ≈ $0.15/1k queries.

LlamaIndex’s TreeSummarize and Compact synthesizers reduce context tokens by 30–50% vs. naive stuffing. LangChain’s map_reduce and refine chains do the same but require more explicit configuration.

Limits and failure modes

LlamaIndex’s docstore/index store can become a write bottleneck during high-throughput ingestion (thousands of docs/sec). The fix is a persistent docstore (Redis, MongoDB) and disabling SimpleDocumentStore. LangChain has no equivalent component — the bottleneck moves entirely to the vector store’s write path.

Both frameworks struggle with incremental updates at scale. LlamaIndex’s refresh_ref_docs compares hashes and updates changed nodes, but it still rewrites vectors for modified chunks. LangChain has no built-in incremental API; you implement upsert logic per store. For true CDC-style updates, you need a vector store with native upsert (Pinecone, Qdrant, Weaviate) and application-level change detection.

Which to choose

Choose LlamaIndex if:

• You want a single abstraction that handles chunking, embedding, indexing, retrieval, reranking, and synthesis with sensible defaults.
• You need portable metadata filters and hybrid search across multiple vector backends.
• Your team prefers configuration over composition — fewer moving parts to wire and debug.
• You’re building a RAG-first application where the index is the central data structure.

Choose LangChain if:

• You’re building an agent or multi-step chain where retrieval is one tool among many (SQL, API, code execution).
• You need fine-grained control over every retrieval parameter and don’t mind writing the glue.
• Your vector store has capabilities the LlamaIndex integration doesn’t expose (custom HNSW params, specialized filters, server-side rerank).
• You’re already invested in the LangChain ecosystem (LangGraph, LangSmith, existing chains).

Choose neither (go direct to the vector store) if:

• You’re at 10M+ vectors and the framework’s coordination layer adds measurable latency or memory pressure.
• You need custom ANN index builds (product quantization, custom distance functions, GPU-accelerated IVF).
• Your ingestion pipeline is a separate Spark/Flink job that writes directly to the store.

At million-vector scale, the vector store choice matters more than the framework. Pick the store first (Pinecone for managed, Qdrant for self-hosted, PGVector if you’re already on Postgres), then pick the framework that gets out of your way. LlamaIndex gets out of the way for RAG; LangChain gets out of the way for agents.