Build a RAG query engine in LlamaIndex with n4n.ai

Retrieval-augmented generation grounds model output in your own data instead of parametric memory. This llamaindex rag query engine tutorial walks through building a working pipeline from ingestion to a query engine that answers questions over a custom corpus, using LlamaIndex and an OpenAI-compatible gateway.

Prerequisites

Python 3.10 or newer
pip and a virtual environment
A small text corpus (we’ll generate one inline)
An API key for an OpenAI-compatible LLM endpoint (we use n4n.ai below)
Familiarity with Python and basic RAG concepts

Install dependencies

Install the core package plus the OpenAI LLM adapter and a local embedding model to avoid external embedding calls.

pip install llama-index llama-index-llms-openai llama-index-embeddings-huggingface

Load and chunk documents

LlamaIndex ingests data as Document objects and splits them into Node objects. For a quick start, write a tiny corpus and load it.

from llama_index.core import SimpleDirectoryReader

# Create a minimal corpus for the demo
with open("corpus.txt", "w") as f:
    f.write("n4n.ai is an LLM inference gateway.\n")
    f.write("It routes to 240+ models with automatic fallback.\n")
    f.write("LlamaIndex builds query engines for RAG.\n")

docs = SimpleDirectoryReader(input_files=["corpus.txt"]).load_data()
print(f"Loaded {len(docs)} documents")

Expected output:

Loaded 1 documents

The default node parser splits on token windows of 1024 tokens. Our file is tiny, so it stays as a single node.

Configure the LLM and embeddings

Set global Settings so every index and engine picks them up. Use a local HuggingFace model for embeddings—no API cost, deterministic vectors.

from llama_index.llms.openai import OpenAI
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import Settings

Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Point LlamaIndex at n4n.ai's OpenAI-compatible endpoint.
# It provides automatic fallback across 240+ models and per-token metering.
Settings.llm = OpenAI(
    model="gpt-4o-mini",
    api_base="https://api.n4n.ai/v1",
    api_key="YOUR_N4N_API_KEY",
    temperature=0.1,
)

If you prefer a different gateway, swap api_base and model. The rest of the tutorial is endpoint-agnostic.

Build the vector index

A VectorStoreIndex embeds each node and stores the vectors in memory. For production, back it with Pinecone, Chroma, or pgvector.

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(docs)
print(f"Index contains {len(index.docstore.docs)} doc nodes")

Expected output:

Index contains 1 doc nodes

Create a query engine

The query engine runs retrieval then synthesis. similarity_top_k controls how many nodes feed the prompt.

query_engine = index.as_query_engine(similarity_top_k=1)

Run a query

Ask a question grounded in the corpus. The engine retrieves the node and passes it to the LLM.

response = query_engine.query("What does n4n.ai do?")
print(str(response))

Expected output (wording may vary):

n4n.ai is an LLM inference gateway that routes to 240+ models with automatic fallback.

This confirms the pipeline works. The remainder of this llamaindex rag query engine tutorial adds production-oriented features.

Filter by metadata

Tag documents with metadata and restrict retrieval to a subset. This cuts noise when your corpus spans multiple domains.

from llama_index.core import Document
from llama_index.core.vector_stores import MetadataFilters, MetadataFilter

docs = [
    Document(text="n4n.ai routes to 240+ models.", metadata={"source": "gateway"}),
    Document(text="LlamaIndex builds RAG query engines.", metadata={"source": "framework"}),
]
index = VectorStoreIndex.from_documents(docs)

filters = MetadataFilters(filters=[MetadataFilter(key="source", value="gateway")])
query_engine = index.as_query_engine(similarity_top_k=1, filters=filters)

print(query_engine.query("What routes to many models?"))

Only the gateway node is eligible, so the answer stays on topic.

Stream tokens

For chat-like UX, stream the synthesis token by token.

query_engine = index.as_query_engine(streaming=True)
streaming_response = query_engine.query("Explain the gateway.")
for token in streaming_response.response_gen:
    print(token, end="")

Customize the prompt

Override the default QA template when you need concise answers or a specific format.

from llama_index.core import PromptTemplate

qa_prompt = PromptTemplate("Context: {context_str}\nQuestion: {query_str}\nAnswer concisely:")

query_engine = index.as_query_engine(text_qa_template=qa_prompt)
print(query_engine.query("What is LlamaIndex?").response)

Persist and reload

Don’t rebuild embeddings on every restart. Persist the storage context to disk.

index.storage_context.persist(persist_dir="./storage")

# Later, in a new process:
from llama_index.core import StorageContext, load_index_from_storage
ctx = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(ctx)

Tune chunk size for recall

The default 1024-token window is arbitrary. For dense technical docs, drop to 512 and raise similarity_top_k to 3–5. For long narrative text, 2048 may work better. Measure with a held-out question set; don’t guess.

from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
index = VectorStoreIndex.from_documents(docs, node_parser=splitter)

Wire up a fallback router

If you skip n4n.ai’s built-in fallback, implement your own in the LLM client. LlamaIndex’s OpenAI class raises on rate limit; catch and retry with a secondary base URL. For most teams, using a gateway that honors client routing directives is simpler than hand-rolling retries.

Recap

You loaded text, embedded it locally, built a VectorStoreIndex, and served queries through a QueryEngine. The llamaindex rag query engine tutorial above is runnable as-is—swap the corpus and model name for your use case. Add metadata filters, streaming, and persistence before shipping to production.