This llamaindex rag agent tutorial walks through building a retrieval-augmented generation agent that answers questions over private docs using GPT-5. We’ll use LlamaIndex’s agent primitives and the OpenAI-compatible chat interface so you can swap models without rewriting code.
Prerequisites
- Python 3.10 or newer
- An OpenAI API key with access to
gpt-5(or a gateway key that fronts it) - A folder of text or PDF files at
./data - Basic familiarity with async Python and
.envfiles
Install the dependencies:
pip install llama-index llama-index-llms-openai llama-index-readers-file python-dotenvCreate a .env with your key:
echo "OPENAI_API_KEY=sk-..." > .envConfigure ingestion and chunking
LlamaIndex splits documents into nodes before embedding. Default chunk size (1024 tokens) works for most prose, but policy PDFs often need smaller chunks to keep citations tight. Set it explicitly.
from llama_index.core import Settings
from llama_index.core.node_parser import SentenceSplitter
Settings.chunk_size = 512
Settings.chunk_overlap = 64
Settings.text_splitter = SentenceSplitter(
chunk_size=512, chunk_overlap=64
)This runs before you build any index. The splitter keeps sentences intact, which matters when the agent later quotes a source.
Build the vector index
Load raw files and embed them. SimpleDirectoryReader handles .txt, .pdf, and common office formats.
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("./data").load_data()
print(f"Loaded {len(documents)} documents")
index = VectorStoreIndex.from_documents(documents)
print("Index built with", len(index.docstore), "nodes")Expected output:
Loaded 3 documents
Index built with 41 nodesThe node count depends on your files. If you see only one node per document, your chunk size is larger than the doc—lower it.
Configure the GPT-5 LLM
Instantiate the OpenAI LLM with gpt-5. LlamaIndex passes kwargs straight through, so we set a low temperature for factual QA.
from llama_index.llms.openai import OpenAI
llm = OpenAI(
model="gpt-5",
temperature=0,
timeout=60,
max_tokens=1024,
)If you want resilience against OpenAI outages, point api_base at an OpenAI-compatible gateway. For example, n4n.ai exposes GPT-5 alongside 240+ models behind one endpoint and applies automatic fallback when a provider is rate-limited or degraded, while still forwarding your cache-control hints.
llm = OpenAI(
model="gpt-5",
api_base="https://api.n4n.ai/v1",
api_key=os.environ["N4N_API_KEY"],
)The rest of the code is identical regardless of which base URL you use.
Wrap the index as a query engine tool
Agents act through tools. Convert the index into a query engine, then wrap it with metadata that tells GPT-5 when to use it.
from llama_index.core.tools import QueryEngineTool, ToolMetadata
query_engine = index.as_query_engine(llm=llm, similarity_top_k=3)
retriever_tool = QueryEngineTool(
query_engine=query_engine,
metadata=ToolMetadata(
name="company_docs",
description="Use this tool to answer questions about internal company docs, pricing, or policy. Input must be a specific question, not a vague phrase.",
),
)similarity_top_k=3 pulls three nodes per query. For a 41-node index this is plenty; for larger corpora, raise it to 5–8.
Build the agent
We use FunctionAgent (the current name for the ReAct-style agent) inside an AgentRunner. The system prompt forces retrieval-only answers.
from llama_index.core.agent import FunctionAgent, AgentRunner
system_prompt = (
"You are a research assistant. Use the company_docs tool to find facts. "
"Never answer from prior knowledge. If the tool returns no relevant context, "
"respond with 'I don't know based on the provided documents.'"
)
agent = AgentRunner(
agent=FunctionAgent(
llm=llm,
tools=[retriever_tool],
system_prompt=system_prompt,
),
)Run a multi-step query
Ask something that requires two retrievals and a comparison:
response = agent.chat(
"What is our refund policy for annual plans, and does it differ from monthly plans?"
)
print(str(response))Expected output (truncated for brevity):
Based on the company docs:
- Annual plans: full refund within 30 days of purchase (source: policy.pdf, p.2)
- Monthly plans: prorated refund on unused days (source: policy.pdf, p.3)
The policies differ: annual plans require a full upfront refund inside a fixed window, while monthly plans are prorated.The agent called company_docs twice—once per plan type—then synthesized. That’s the core value of a llamaindex rag agent tutorial: the loop decides tool calls for you.
Inspect the trace
Set verbose=True to see the reasoning steps during development:
agent.verbose = True
response = agent.chat("Summarize the security section of the handbook")You’ll get logs similar to:
Thought: I need to find the security section.
Action: company_docs
Action Input: security section of handbook
Observation: The handbook states MFA is required for all prod access...
Thought: I have enough to summarize.If you see the agent calling the tool with the exact user question verbatim, tighten the tool description.
Production considerations
A RAG agent in production needs more than a happy path.
Cache embeddings and LLM calls
Re-embedding static docs on every boot wastes tokens. Persist the index to disk:
index.storage_context.persist(persist_dir="./storage")For LLM responses, attach a cache to the model:
from llama_index.core.cache import RedisCache
llm.cache = RedisCache(host="localhost", port=6379)This avoids repeated GPT-5 charges for identical agent prompts.
Token metering
When you route through a gateway, per-token usage metering lets you attribute cost per session. The OpenAI client returns usage on each call:
raw = llm.chat([ChatMessage(role="user", content="hi")])
print(raw.additional_kwargs["usage"])Log this per agent.chat invocation if you bill customers by usage.
Fallback and routing
A ReAct loop fails hard if GPT-5 returns 429 mid-reasoning. An OpenAI-compatible endpoint that honors client routing directives and fronts multiple providers prevents that. Your code stays the same; only api_base changes.
Evaluate the agent
Add a smoke test so regressions surface in CI:
def test_refund_question():
resp = agent.chat("What is the refund window for annual plans?")
assert "30 days" in str(resp)
assert "policy.pdf" in str(resp)Run with pytest. If the agent starts hallucinating, the test catches it before users do.
Where to go next
This llamaindex rag agent tutorial covered the minimum viable retrieval agent. Extend it by adding a router tool that selects between multiple indexes (e.g., engineering vs. legal), or replace FunctionAgent with the LlamaIndex workflow API when you need explicit state transitions. Keep the system prompt strict, log every tool call, and treat GPT-5 as a remote service that will occasionally fail.