LangChain vs LlamaIndex: agent support inside RAG

If you’re building a RAG system that needs to reason, not just retrieve, you’ve hit the agent question. Both LangChain and LlamaIndex support agents, but they model the problem differently. LangChain treats agents as a general-purpose control loop over arbitrary tools. LlamaIndex treats agents as query engines with tool use baked into the retrieval path. That distinction shapes everything: how you compose retrieval, how you debug, and where the latency hides.

Architecture philosophy

LangChain’s AgentExecutor is a thin loop: LLM decides action → tool runs → observation returns → repeat. The agent doesn’t know it’s doing RAG. It just sees a retriever tool alongside calculator, search, and sql_db. This makes LangChain agents flexible for multi-hop workflows that mix retrieval with computation, API calls, or browser automation.

LlamaIndex’s AgentRunner (and the newer FunctionCallingAgentWorker) wraps a QueryEngine that already understands indexes, retrievers, and response synthesis. The agent is a query engine that can call tools. Retrieval isn’t a tool you bolt on; it’s the primary path, with tools as escape hatches.

# LangChain: retrieval is one tool among many
from langchain.agents import create_openai_functions_agent, AgentExecutor
from langchain.tools.retriever import create_retriever_tool

retriever_tool = create_retriever_tool(
    retriever=vectorstore.as_retriever(k=4),
    name="policy_docs",
    description="Search company policy documents"
)
tools = [retriever_tool, calculator_tool, http_tool]

agent = create_openai_functions_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, max_iterations=5)
result = executor.invoke({"input": "What's the travel budget for London?"})

# LlamaIndex: retrieval is the backbone, tools extend it
from llama_index.core.agent import FunctionCallingAgentWorker
from llama_index.core.tools import QueryEngineTool, FunctionTool
from llama_index.core.query_engine import RetrieverQueryEngine

query_engine = RetrieverQueryEngine.from_args(
    retriever=index.as_retriever(similarity_top_k=4),
    response_mode="compact"
)
tools = [
    QueryEngineTool.from_defaults(
        query_engine=query_engine,
        name="policy_docs",
        description="Company policy lookup"
    ),
    FunctionTool.from_defaults(fn=calculate_budget, name="calculator")
]

agent = FunctionCallingAgentWorker.from_tools(tools, llm=llm).as_agent()
response = agent.chat("What's the travel budget for London?")

Retrieval composition

LangChain gives you RetrieverTool and VectorStoreRetriever. You can chain retrievers with EnsembleRetriever or ParentDocumentRetriever, but composition happens outside the agent. The agent sees one tool that returns documents. If you need hybrid search + reranking + recursive retrieval, you build that pipeline first, then expose it as a single tool.

LlamaIndex builds composition into the query engine hierarchy. SubQuestionQueryEngine decomposes a query, routes sub-questions to different indexes, and synthesizes. RouterQueryEngine picks between vector, keyword, and SQL indexes. RecursiveRetriever walks hierarchies. The agent can call these as tools, but you often don’t need an agent at all — the query engine is the planner.

# LlamaIndex: multi-index routing without an agent
from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import LLMSingleSelector
from llama_index.core.tools import QueryEngineTool

vector_tool = QueryEngineTool.from_defaults(
    query_engine=vector_index.as_query_engine(),
    description="Semantic search over policy PDFs"
)
sql_tool = QueryEngineTool.from_defaults(
    query_engine=sql_index.as_query_engine(),
    description="Structured queries over employee table"
)

router = RouterQueryEngine(
    selector=LLMSingleSelector.from_defaults(),
    query_engine_tools=[vector_tool, sql_tool]
)
response = router.query("How many engineers in London and what's their travel policy?")

Memory and state

LangChain’s AgentExecutor accepts a memory parameter (e.g., ConversationBufferMemory, ConversationSummaryMemory). The memory stores full message history and injects it into the agent prompt each turn. Simple, but the agent doesn’t distinguish between “conversation context” and “retrieved context” — both jam into the same context window.

LlamaIndex separates concerns. ChatMemoryBuffer manages conversation history with token limits and summarization. ChatStore persists across sessions. The query engine handles retrieved context independently. An agent can use memory, but the query engine path doesn’t require it.

# LangChain: memory injected into agent prompt
from langchain.memory import ConversationSummaryBufferMemory

memory = ConversationSummaryBufferMemory(
    llm=llm, max_token_limit=2000, return_messages=True
)
executor = AgentExecutor(agent=agent, tools=tools, memory=memory)

# LlamaIndex: memory on the agent, separate from retrieval
from llama_index.core.memory import ChatMemoryBuffer

memory = ChatMemoryBuffer.from_defaults(token_limit=3000)
agent = FunctionCallingAgentWorker.from_tools(
    tools, llm=llm, memory=memory
).as_agent()

Streaming and intermediate steps

LangChain streams via astream_events or astream_log. You get on_chain_start, on_tool_start, on_tool_end, on_chain_end events. Reconstructing a readable trace requires filtering. The AgentExecutor returns intermediate_steps as a list of (AgentAction, observation) tuples — useful for debugging, awkward for UI.

LlamaIndex streams via astream_chat on the agent, yielding ChatResponse deltas. The AgentChatResponse includes sources (retrieved nodes) and tool_calls with structured input/output. The newer FunctionCallingAgentWorker emits AgentStreamDelta with typed fields. Cleaner for building streaming UIs.

# LangChain: event stream requires assembly
async for event in executor.astream_events({"input": query}, version="v2"):
    if event["event"] == "on_tool_end":
        print(f"Tool {event['name']} returned: {event['data']['output']}")
    elif event["event"] == "on_chat_model_stream":
        print(event["data"]["chunk"].content, end="")

# LlamaIndex: typed streaming deltas
async for delta in agent.astream_chat("What's the travel budget?"):
    if delta.response:
        print(delta.response, end="")
    if delta.tool_calls:
        for tc in delta.tool_calls:
            print(f"\n[Tool: {tc.tool_name}] {tc.tool_input}")

Ecosystem and integrations

LangChain wins on breadth. If you need a tool for Notion, Jira, Slack, Apify, or a custom REST endpoint, someone has published a langchain-community tool. The Tool interface is trivial to implement. You can drop into the agent loop with minimal ceremony.

LlamaIndex’s tool ecosystem is smaller but more retrieval-focused. Tools like QueryEngineTool, RetrieverTool, and SQLTableRetrieverQueryEngine are first-class. Custom tools use FunctionTool.from_defaults(fn=...). The gap narrows if your tools are primarily data access, widens if you need SaaS integrations.

Latency and token economics

LangChain agents burn tokens on the planning loop. Each iteration: system prompt + history + tool schemas + previous observations → LLM → action. A 3-hop query easily hits 4-6 LLM calls. You pay for the planner and the tool calls.

LlamaIndex query engines often solve the same problem in 1-2 calls: route → retrieve → synthesize. The agent path adds planner overhead only when tools are actually invoked. For pure RAG, LlamaIndex is cheaper and faster. For mixed tool/RAG workloads, the gap narrows.

When to choose which

Choose LangChain when:

• Your agent spends more time calling APIs, browsers, or calculators than retrieving documents.
• You need off-the-shelf tools for SaaS products (Notion, Salesforce, Jira, Zapier).
• You want a single agent framework that also handles non-RAG chains (extraction, classification, summarization).
• Your team already standardizes on LangChain primitives (Runnable, LCEL, LangGraph).

Choose LlamaIndex when:

• Retrieval is the core workload — hybrid search, reranking, recursive retrieval, SQL+vector fusion.
• You need multi-index routing or sub-question decomposition without an agent loop.
• You want structured output from retrieval (Pydantic models, SQL rows) that feeds directly into synthesis.
• You’re building a chat-over-data product where conversation memory and citation fidelity matter.

Consider neither when:

• Your RAG is single-index, single-hop, no tools. A raw RetrieverQueryEngine or create_retrieval_chain is simpler.
• You need deterministic control flow. Use LangGraph or LlamaIndex Workflow (beta) for explicit state machines.
• You’re serving high-throughput, low-latency inference. Both frameworks add overhead; a hand-rolled loop with n4n.ai or direct provider SDKs wins on tail latency.

The pragmatic middle

Most production systems I’ve seen settle on a hybrid: LlamaIndex for the retrieval backbone (indexing, routing, recursive retrieval, structured query engines), LangChain tools for the long tail of SaaS integrations, and a thin orchestration layer (often LangGraph or custom) that decides when to call which. The frameworks aren’t mutually exclusive — they solve adjacent problems. Pick the one that matches your primary workload, wrap the other for the edges.