Building a RAG pipeline in LlamaIndex with n4n.ai

Building a llamaindex rag pipeline n4n.ai integration is straightforward because the gateway exposes an OpenAI-compatible endpoint, so you can reuse LlamaIndex’s built-in providers without writing custom adapters. This tutorial builds a local document Q&A system that indexes text files and answers questions using a hosted model through that single endpoint.

Prerequisites

• Python 3.10 or newer
• llama-index and openai packages
• An API key for the gateway (set as N4N_API_KEY in your environment)
• A ./data directory containing a few .txt or .md files to index
• Familiarity with Python virtual environments

If you already have a LlamaIndex project, skip the install and adapt the Settings block.

Step 1: Install dependencies

pip install "llama-index" openai

LlamaIndex splits into core and provider packages in recent versions; the meta package pulls what we need. Create a fresh virtual environment to avoid version conflicts with other LLM tooling.

Step 2: Configure the LLM and embeddings

LlamaIndex uses a global Settings object to control which LLM and embedding model run the pipeline. Point both at the OpenAI-compatible base URL:

import os
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index import Settings

Settings.llm = OpenAI(
    model="openai/gpt-4o-mini",
    api_key=os.environ["N4N_API_KEY"],
    base_url="https://api.n4n.ai/v1",
    temperature=0.1,
)
Settings.embeddings = OpenAIEmbedding(
    model="openai/text-embedding-3-small",
    api_key=os.environ["N4N_API_KEY"],
    base_url="https://api.n4n.ai/v1",
)

The model string follows the gateway’s routing convention. You can swap openai/gpt-4o-mini for any of the 240+ addressed models, such as anthropic/claude-3-haiku, without changing client code. Because the endpoint honors client routing directives, the same OpenAI class works unchanged.

Step 3: Load and chunk documents

Use SimpleDirectoryReader for plain files. Tune chunk_size and chunk_overlap on Settings to match your model’s context window:

from llama_index import VectorStoreIndex, SimpleDirectoryReader

Settings.chunk_size = 512
Settings.chunk_overlap = 64

docs = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(docs)

Expected console output during indexing:

INFO:llama_index.core:Parsing nodes: 100%|██████████| 12/12 [00:00<00:00, 2000.0it/s]
INFO:llama_index.core:Generating embeddings: 100%|██████████| 12/12 [00:01<00:00, 8.5it/s]

You now have an in-memory vector index. For production, persist it with index.storage_context.persist() and a real vector store. Chunk size is the lever most teams ignore. With text-embedding-3-small, 512-token chunks balance retrieval precision and latency. Smaller chunks improve recall on specific facts but increase query cost.

If your documents have structure (headings, tables), use SentenceSplitter with paragraph_separator instead of the default tokenizer:

from llama_index.core.node_parser import SentenceSplitter

Settings.text_splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)

Step 4: Query the index

Build a query engine and run a synchronous question:

query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("What is the refund policy for annual plans?")
print(str(response))

A typical response:

Based on the provided documents, annual plans are refunded within 30 days of purchase if no more than 10% of the subscription period has been used. After that, prorated refunds are not offered.

The retrieved nodes are accessible via response.source_nodes if you need to show citations. Always log response.metadata in development to inspect token usage. Print sources during debugging:

for node in response.source_nodes:
    print(node.metadata.get("file_name"), node.score)

Step 5: Streaming and chat

For chat-style UIs, use as_chat_engine with streaming:

chat_engine = index.as_chat_engine(streaming=True)
stream = chat_engine.stream_chat("Summarize the onboarding steps.")
for token in stream.response_gen:
    print(token, end="")

The LlamaIndex RAG pipeline stays identical regardless of which backend model serves the request. Behind the gateway, automatic fallback engages when a provider is rate-limited or degraded, so the OpenAI client sees a single stable endpoint. That removes the need to write retry loops in your application code.

If you want to forward provider cache-control hints, pass them through the gateway’s headers; the endpoint forwards them unchanged. Per-token usage metering appears in the API response headers, so you can reconcile cost without custom instrumentation.

Step 6: Metadata filtering

Real corpora need scoping. Attach metadata at load time and filter at query time:

from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter

docs = SimpleDirectoryReader(
    "./data",
    file_metadata=lambda path: {"category": "legal" if "legal" in path else "product"}
).load_data()

index = VectorStoreIndex.from_documents(docs)

filters = MetadataFilters(filters=[ExactMatchFilter(key="category", value="legal")])
query_engine = index.as_query_engine(filters=filters, similarity_top_k=2)

This restricts retrieval to legal documents before the LLM sees context, cutting hallucination surface.

Step 7: Persist and scale

In-memory indexes die with the process. Persist to disk:

index.storage_context.persist(persist_dir="./storage")

Reload later:

from llama_index.core import StorageContext, load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

For multi-worker deployments, back the index with pgvector or Qdrant. The Settings LLM configuration above remains the same; only the storage context changes.

Production notes

Set temperature=0 for factual QA. Higher values add hallucination risk with no retrieval benefit. Evaluate recall with a held-out question set before shipping—measure whether the correct node appears in similarity_top_k=3.

LlamaIndex’s default VectorStoreIndex uses in-memory storage. Swap to a real vector store via storage_context for concurrent access. The LLM configuration above remains unchanged across providers.

The gateway’s per-token metering lets you attribute cost per request without proxy middleware. Capture response.metadata or response headers in your logging layer.

That is a complete, runnable LlamaIndex RAG pipeline against an OpenAI-compatible gateway. Modify the model string to experiment with different providers without touching your indexing or query code.