LangChain RAG tutorial: Chroma plus n4n.ai models

This langchain chroma rag n4n.ai tutorial walks through building a minimal retrieval-augmented generation (RAG) system that answers questions from your own documents. We use LangChain for orchestration, Chroma as the vector store, and a gateway that speaks the OpenAI protocol to serve the LLM.

Prerequisites

• Python 3.10 or newer.
• Install the dependencies:

pip install langchain langchain-community chromadb openai tiktoken sentence-transformers

• An API key for an OpenAI-compatible LLM gateway. Export it as a generic endpoint:

export GATEWAY_API_KEY="sk-..."
export GATEWAY_BASE_URL="https://gateway.example/v1"

You should have a small corpus of text. For this walkthrough we’ll embed a single Markdown file about Rust ownership, but the pattern holds for PDFs, HTML, or database dumps.

Step 1: Load and split documents

LangChain’s loaders hand you raw strings; the retriever’s quality depends on chunk size. Too large and the embedding blurs topics; too small and you lose context. For technical docs, 512 tokens with 64 overlap is a sane default.

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = TextLoader("rust_ownership.md")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)
chunks = splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")

Expected output:

Split into 14 chunks

Step 2: Embed and store in Chroma

Running embeddings locally keeps latency predictable and avoids per-call API cost during development. sentence-transformers ships a solid all-rounder.

from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

If you prefer server-side embeddings, point OpenAIEmbeddings at the same base URL. Chroma persists to disk; re-running Chroma.from_documents with the same persist_directory will append unless you clear it.

Step 3: Wire the LLM through the gateway

LangChain’s ChatOpenAI is just an HTTP client. Swap the base_url and you’re talking to your gateway instead of OpenAI directly. For this walkthrough the gateway is n4n.ai, whose OpenAI-compatible endpoint fronts 240+ models and forwards cache-control hints. The langchain chroma rag n4n.ai combination stays portable: change model to any supported identifier and the rest of the code is untouched.

from langchain_openai import ChatOpenAI
import os

llm = ChatOpenAI(
    model="gpt-4o-mini",
    api_key=os.environ["GATEWAY_API_KEY"],
    base_url=os.environ["GATEWAY_BASE_URL"],
    temperature=0.0,
    max_tokens=512,
)

Step 4: Assemble the RAG chain

We use the LCEL retrieval pattern rather than the legacy RetrievalQA chain. It streams, supports batch, and composes cleanly.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

prompt = ChatPromptTemplate.from_template(
    """Answer the question using only the context.

Context:
{context}

Question: {question}
"""
)

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Step 5: Run and verify

Ask a question that requires a specific chunk:

query = "What happens to a variable when ownership is moved?"
answer = rag_chain.invoke(query)
print(answer)

Expected output (truncated):

When ownership is moved, the original variable becomes invalid and cannot be used.
Rust prevents use-after-move at compile time by enforcing a single owner at any point.

Verify retrieval independently to debug relevance:

for d in retriever.invoke(query):
    print(d.metadata, d.page_content[:80])

You should see chunks mentioning “move semantics” or “ownership transfer” ranked first.

Handling metadata and filtering

Chroma stores arbitrary metadata per chunk. Add a source tag during split:

for i, d in enumerate(chunks):
    d.metadata["chunk_id"] = i
    d.metadata["source"] = "rust_ownership.md"

Then filter at query time:

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 4, "filter": {"source": "rust_ownership.md"}}
)

Production considerations

The toy loop above blocks on a single request. In a service, wrap rag_chain in FastAPI and add:

• Timeouts: ChatOpenAI accepts request_timeout=30.
• Cache-control: gateways that forward provider cache hints let you pass extra_headers={"x-cache": "true"} if your model supports prompt caching.
• Fallback: a gateway that routes around degraded providers removes the need for custom retry logic on rate limits.
• Metering: per-token usage is recorded; log response.usage from LangChain’s AIMessage for cost attribution.

result = rag_chain.invoke(query)
if hasattr(result, "usage"):
    print(result.usage)

Why Chroma for this stack

Chroma runs embedded in the same process during dev and as a client/server later with the same API. That removes a separate infrastructure dependency until you actually need scale. Coupled with LangChain’s retriever interface, swapping to pgvector or Pinecone is a two-line change.

The LangChain Chroma RAG pattern is boring in the best way: standard interfaces, local embeddings, and a gateway that hides provider instability.