Migrating a RAG app from LlamaIndex to LangChain

Migrating a RAG application from LlamaIndex to LangChain is a common architectural shift as teams standardize on LangChain’s broader ecosystem for agents and tooling. The two frameworks share similar concepts but differ significantly in API design, document processing pipelines, and retrieval abstractions. This guide walks through a complete migration with working code at each stage so you can verify parity before cutting over.

Step 1: inventory your LlamaIndex components

Before writing any LangChain code, map your existing LlamaIndex pipeline. A typical RAG app has four moving parts: document ingestion and chunking, embedding model selection, vector store configuration, and the query engine with its retriever and response synthesizer. Open your codebase and note each class instantiation and parameter.

# llamaindex_pipeline.py — what you likely have today
from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    Settings,
    get_response_synthesizer,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

# Global settings (LlamaIndex pattern)
Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.node_parser = SentenceSplitter(chunk_size=1024, chunk_overlap=128)

# Vector store
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("docs")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

# Ingestion
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents, vector_store=vector_store)

# Query engine
retriever = index.as_retriever(similarity_top_k=4)
response_synthesizer = get_response_synthesizer(response_mode="compact")
query_engine = index.as_query_engine(
    retriever=retriever,
    response_synthesizer=response_synthesizer,
)

# Runtime query
response = query_engine.query("What is the refund policy?")
print(response)

Capture the chunk size, overlap, embedding model, top-k, and response mode. These are your parity targets.

Step 2: set up the LangChain equivalents

LangChain separates concerns into distinct modules: document loaders, text splitters, embedding models, vector stores, and chains. Install the minimal dependency set first.

pip install langchain langchain-openai langchain-community chromadb

Now replicate each LlamaIndex component. Note that LangChain uses explicit composition over global settings.

# langchain_pipeline.py — parity implementation
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# 1. Document loading (equivalent to SimpleDirectoryReader)
loader = DirectoryLoader("./data", glob="**/*.txt", loader_cls=TextLoader)
documents = loader.load()

# 2. Text splitting (equivalent to SentenceSplitter)
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1024,
    chunk_overlap=128,
    separators=["\n\n", "\n", " ", ""],
)
splits = text_splitter.split_documents(documents)

# 3. Embeddings (equivalent to OpenAIEmbedding)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# 4. Vector store (equivalent to ChromaVectorStore)
vectorstore = Chroma.from_documents(
    documents=splits,
    embedding=embeddings,
    persist_directory="./chroma_db",
    collection_name="docs",
)

# 5. Retriever (equivalent to index.as_retriever)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

# 6. LLM (equivalent to OpenAI LLM)
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# 7. Prompt template (replaces response_synthesizer)
prompt = ChatPromptTemplate.from_template("""Answer the question based only on the following context:
{context}

Question: {question}
""")

# 8. Chain (replaces query_engine)
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Runtime query
response = rag_chain.invoke("What is the refund policy?")
print(response)

Verify success: run both pipelines against the same five questions. Compare latency, token usage, and answer quality. The responses should be semantically equivalent — minor wording differences are expected.

Step 3: migrate advanced retrieval patterns

LlamaIndex’s VectorStoreIndex supports hybrid search, metadata filtering, and recursive retrieval out of the box. LangChain pushes these to the vector store layer or retriever composition. Here are the three most common patterns.

Hybrid search (vector + keyword)

# LlamaIndex: VectorStoreIndex with hybrid=True
# LangChain: use Chroma's built-in hybrid or ensemble retriever
from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

# BM25 requires the raw splits from Step 2
bm25_retriever = BM25Retriever.from_documents(splits)
bm25_retriever.k = 4

ensemble_retriever = EnsembleRetriever(
    retrievers=[vectorstore.as_retriever(search_kwargs={"k": 4}), bm25_retriever],
    weights=[0.6, 0.4],
)

# Swap into chain
rag_chain_hybrid = (
    {"context": ensemble_retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Metadata filtering

# LlamaIndex: MetadataFilters in retriever
# LangChain: pass filter to search_kwargs
filtered_retriever = vectorstore.as_retriever(
    search_kwargs={
        "k": 4,
        "filter": {"source": {"$eq": "policy_docs/refund.txt"}},
    }
)

Multi-query retrieval (query expansion)

# LlamaIndex: QueryRewriteRetriever or similar
# LangChain: MultiQueryRetriever
from langchain.retrievers.multi_query import MultiQueryRetriever

multi_query_retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    llm=llm,
)

rag_chain_multi = (
    {"context": multi_query_retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Verify success: add integration tests that assert filtered results only contain documents matching the metadata constraint, and that multi-query returns more diverse chunks than the base retriever.

Step 4: handle streaming and citations

Production RAG apps need streaming responses and source attribution. LlamaIndex’s query_engine.query() returns a streaming response object with source nodes attached. LangChain handles these separately.

Streaming

# LlamaIndex: response.response_gen (generator)
# LangChain: chain.astream() or chain.stream()
for chunk in rag_chain.stream("What is the refund policy?"):
    print(chunk, end="", flush=True)
print()

Citations / source documents

# LlamaIndex: response.source_nodes
# LangChain: return source documents alongside answer
from langchain_core.runnables import RunnableParallel

rag_with_sources = RunnableParallel(
    {"answer": rag_chain, "sources": retriever}
)

result = rag_with_sources.invoke("What is the refund policy?")
print("Answer:", result["answer"])
print("Sources:", [doc.metadata for doc in result["sources"]])

For inline citations, use a prompt that forces the model to reference doc IDs, then post-process.

citation_prompt = ChatPromptTemplate.from_template("""Answer the question using only the context below.
Cite sources inline using [doc_N] where N is the document index.

Context:
{context}

Question: {question}
""")

def format_docs_with_ids(docs):
    return "\n\n".join(f"[doc_{i}] {doc.page_content}" for i, doc in enumerate(docs))

citation_chain = (
    {"context": retriever | format_docs_with_ids, "question": RunnablePassthrough()}
    | citation_prompt
    | llm
    | StrOutputParser()
)

Verify success: stream a long answer and confirm tokens arrive incrementally. Check that every citation in the output maps to a retrieved document.

Step 5: migrate evaluation and observability

LlamaIndex has built-in evaluation modules (FaithfulnessEvaluator, RelevancyEvaluator). LangChain delegates to LangSmith or custom evaluators. If you used LlamaIndex evaluators, port them to LangChain’s evaluation API.

# LlamaIndex evaluator pattern
# from llama_index.core.evaluation import FaithfulnessEvaluator
# evaluator = FaithfulnessEvaluator(llm=Settings.llm)
# eval_result = evaluator.evaluate_response(response=response)

# LangChain + LangSmith pattern
from langsmith.evaluation import evaluate
from langsmith.schemas import Run, Example

def faithfulness_evaluator(run: Run, example: Example) -> dict:
    # Custom implementation or use LangSmith's built-in evaluators
    # See: https://docs.smith.langchain.com/evaluation/how_to_guides/faithfulness
    pass

# Or use RAGAS via LangChain integration
# pip install ragas
from ragas import evaluate as ragas_evaluate
from ragas.metrics import faithfulness, answer_relevancy
from datasets import Dataset

# Prepare dataset format for RAGAS
eval_data = {
    "question": ["What is the refund policy?"],
    "answer": [result["answer"]],
    "contexts": [[doc.page_content for doc in result["sources"]]],
    "ground_truth": ["Full refund within 30 days with receipt."],
}
dataset = Dataset.from_dict(eval_data)

# Run evaluation
ragas_result = ragas_evaluate(dataset, metrics=[faithfulness, answer_relevancy])
print(ragas_result)

Verify success: run your evaluation suite against both the old and new pipelines. Faithfulness and relevancy scores should be within 2-3% of each other.

Step 6: address async and concurrency

LlamaIndex’s query engine is synchronous by default with optional async methods (aquery). LangChain’s LCEL runnables are async-native. If your production service uses FastAPI or similar, migrate to async end-to-end.

# FastAPI endpoint — LlamaIndex style (sync)
# @app.post("/query")
# def query(request: QueryRequest):
#     return {"answer": str(query_engine.query(request.question))}

# FastAPI endpoint — LangChain async style
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class QueryRequest(BaseModel):
    question: str

@app.post("/query")
async def query(request: QueryRequest):
    answer = await rag_chain.ainvoke(request.question)
    return {"answer": answer}

Verify success: load test both endpoints with locust or hey. The async LangChain version should handle higher concurrent requests with lower memory per request.

Step 7: cut over with feature flags

Don’t flip a single switch. Deploy the LangChain pipeline behind a feature flag, shadow traffic, and compare production metrics.

# feature_flag_router.py
import os
from contextlib import asynccontextmanager

USE_LANGCHAIN = os.getenv("USE_LANGCHAIN", "false").lower() == "true"

@asynccontextmanager
async def get_rag_chain():
    if USE_LANGCHAIN:
        yield rag_chain  # from Step 2
    else:
        # Wrap LlamaIndex query_engine in compatible interface
        class LlamaIndexWrapper:
            async def ainvoke(self, question: str) -> str:
                return str(query_engine.query(question))
        yield LlamaIndexWrapper()

# In your endpoint
@app.post("/query")
async def query(request: QueryRequest):
    async with get_rag_chain() as chain:
        answer = await chain.ainvoke(request.question)
    return {"answer": answer}

Monitor these metrics for 48-72 hours:

• P50/P99 latency
• Error rate (timeouts, provider errors)
• Token consumption per request
• User feedback scores (thumbs up/down)

If metrics are at parity or better, flip the flag to 100%. Keep the LlamaIndex code for one release cycle as a rollback path.

Step 8: clean up and document

Remove LlamaIndex dependencies from requirements.txt and pyproject.toml. Delete unused imports and the old pipeline module. Update your architecture decision record (ADR) with the migration rationale, parity test results, and any open differences.

# ADR-0042: Migrate RAG pipeline from LlamaIndex to LangChain

## Decision
Standardize on LangChain for all LLM orchestration.

## Rationale
- Unified agent/tool ecosystem for upcoming multi-step workflows
- Native async support reduces thread pool complexity
- LangSmith integration simplifies production observability
- Team familiarity: 3/4 engineers have deeper LangChain experience

## Parity verification
- 50 golden-set questions: 94% semantic equivalence (LLM-as-judge)
- Faithfulness: LlamaIndex 0.87 → LangChain 0.85
- Answer relevancy: LlamaIndex 0.91 → LangChain 0.90
- P99 latency: 2.1s → 1.8s (async benefit)

## Open differences
- LangChain's RecursiveCharacterTextSplitter produces 3% more chunks on average
- Metadata filtering syntax differs; documented in `docs/retrieval-patterns.md`

Common migration pitfalls

When to stay on LlamaIndex

Migration isn’t always the right call. Keep LlamaIndex if:

• Your team has deep LlamaIndex expertise and no agent roadmap
• You rely on LlamaIndex-specific features like KnowledgeGraphIndex or PropertyGraphIndex
• You use LlamaParse for complex PDF extraction and want the native integration
• The cost of migration exceeds the projected maintenance burden of dual-framework support

The migration is mechanical once you map the abstractions. The real work is verification — golden-set evaluation, shadow traffic, and load testing. Treat the cutover as a reliability exercise, not a feature launch, and you’ll avoid the subtle regressions that appear weeks later.