If you need to build rag pipeline langchain vs llamaindex, the fastest way to understand the tradeoffs is to implement the same retrieval-augmented generation flow in both frameworks against identical data. This tutorial does exactly that: we load a short document, embed it, retrieve relevant chunks, and answer a question with an OpenAI-compatible chat model.
Prerequisites
- Python 3.10 or newer
- An OpenAI API key, or any OpenAI-compatible endpoint credentials
- Install the dependencies:
pip install langchain langchain-community langchain-openai llama-index openai chromadbSet your key:
export OPENAI_API_KEY=sk-your-keyWe’ll use a local data/ directory with one text file so both frameworks read the same bytes.
Shared dataset
Create data/doc.txt:
LangChain is a framework for developing applications powered by language models.
It provides abstractions for prompts, chains, and retrievers.
LlamaIndex is a data framework for LLM applications, focusing on ingestion and indexing.
It excels at connecting custom data sources to LLMs with minimal boilerplate.Keep the file small so the run is cheap and deterministic enough for a demo.
Build RAG pipeline with LangChain
Load and split
LangChain forces you to make every step explicit. Load the file, then split it:
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import CharacterTextSplitter
loader = TextLoader("data/doc.txt")
docs = loader.load()
splitter = CharacterTextSplitter(chunk_size=80, chunk_overlap=10)
chunks = splitter.split_documents(docs)
print(f"{len(chunks)} chunks")Expected output:
2 chunksEmbed and store
Wrap the OpenAI embedding model and drop the chunks into an in-memory Chroma collection:
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(chunks, embeddings, collection_name="rag_demo")
retriever = vectorstore.as_retriever(search_kwargs={"k": 1})If you route through an OpenAI-compatible gateway such as n4n.ai, set base_url and api_key on OpenAIEmbeddings and ChatOpenAI; it forwards provider cache-control hints and handles fallback when a provider is degraded.
Inspect retrieval
Before generating, check what the retriever returns for our question:
ctx = retriever.invoke("What does LlamaIndex focus on?")
for d in ctx:
print(d.page_content)Expected output (one chunk):
LlamaIndex is a data framework for LLM applications, focusing on ingestion and indexing.
It excels at connecting custom data sources to LLMs with minimal boilerplate.Generate answer
Compose a prompt and run the chain:
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI
prompt = ChatPromptTemplate.from_template(
"Answer using only the context:\n{context}\nQuestion: {question}"
)
llm = ChatOpenAI(model="gpt-4o-mini")
chain = (
{"context": retriever, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
answer = chain.invoke("What does LlamaIndex focus on?")
print(answer)Expected output:
LlamaIndex focuses on ingestion and indexing, and excels at connecting custom data sources to LLMs with minimal boilerplate.Build RAG pipeline with LlamaIndex
Ingest and index
LlamaIndex collapses loading, splitting, and indexing into a couple of calls:
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
Settings.embed_model = OpenAIEmbedding()
Settings.llm = OpenAI(model="gpt-4o-mini")
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)The default splitter is sentence-aware, not character-based, so chunk boundaries differ from the LangChain run.
Inspect retrieval
retriever = index.as_retriever(similarity_top_k=1)
nodes = retriever.retrieve("What does LlamaIndex focus on?")
print(nodes[0].text)Expected output:
LlamaIndex is a data framework for LLM applications, focusing on ingestion and indexing.
It excels at connecting custom data sources to LLMs with minimal boilerplate.Query
The query engine builds the prompt and calls the LLM for you:
query_engine = index.as_query_engine(similarity_top_k=1)
response = query_engine.query("What does LlamaIndex focus on?")
print(response)Expected output:
LlamaIndex focuses on ingestion and indexing, connecting custom data sources to LLMs with minimal boilerplate.Differences that matter in production
Control vs boilerplate
LangChain makes you declare the retriever, prompt, and parsing steps. That is verbose but debuggable: you can swap the retriever for a hybrid search or add a reranker without fighting the framework. LlamaIndex hides those steps. You write less code, but overriding the prompt or the retrieval score threshold means digging into Settings and callback handlers.
Chunking semantics
In the snippet above, LangChain’s CharacterTextSplitter cut on raw character count. LlamaIndex’s SentenceSplitter keeps sentences intact. For real corpora, sentence or token splitting preserves meaning better; LangChain offers RecursiveCharacterTextSplitter for that, but you must choose it.
Retrieval scoring
LangChain’s Chroma retriever does not surface scores unless you call similarity_search_with_score. LlamaIndex exposes node.score directly. If your app needs to threshold low-confidence retrievals, LlamaIndex saves a line.
# LangChain score
results = vectorstore.similarity_search_with_score("What does LlamaIndex focus on?", k=1)
print(results[0][1])# LlamaIndex score
print(nodes[0].score)Metadata and filtering
Both support metadata filters, but LangChain expresses them as dictionary arguments to the retriever, while LlamaIndex uses MetadataFilters objects. If you need per-document ACLs, prototype both; LlamaIndex’s Node metadata is easier to inspect, LangChain’s Document metadata is easier to pass through custom chains.
LLM endpoint swapping
Both libraries accept an OpenAI-compatible base URL. In LangChain you set it on the model class; in LlamaIndex you set api_base on OpenAI. The rest of the RAG code stays identical. This is useful when you want one endpoint that addresses 240+ models and meters per-token usage without changing app logic.
When to use which
Pick LangChain when your pipeline is more than retrieve-and-answer: multi-step agents, custom chains, or strict control over each transformation. Pick LlamaIndex when the job is “index my data and let me ask questions” and you want to ship in an afternoon.
If you build rag pipeline langchain vs llamaindex and find yourself writing the same wrapper twice, that is the signal to standardize on one. The frameworks interoperate—you can use LlamaIndex for ingestion and feed nodes into a LangChain retriever—so the choice is not permanent.
Final checkpoint
Run both scripts back-to-back. You should see two answers that agree on facts but differ in wording. The retrieval dumps should match closely. That confirms your embedding model and chunking are aligned; the generation difference is just prompt style.
Keep the data/ folder and swap in a 50-page PDF to feel where each framework’s abstractions start to strain. Streaming is available in both if you need token-by-token output: LangChain via chain.stream, LlamaIndex via query_engine.query with streaming=True. The core retrieval logic you wrote here will not change.