Building a personal AI assistant on your data with RAG

A personal AI assistant RAG own data stack lets you query private notes, code, and docs without shipping them to a third-party trainer. This tutorial builds a minimal, runnable retrieval-augmented generation loop in Python that embeds locally and calls an OpenAI-compatible model for answers.

Prerequisites

• Python 3.11+ and a virtual environment.
• A folder of plain-text or Markdown files you want to query (e.g., ~/notes).
• API key for any OpenAI-compatible inference endpoint. If you want one endpoint that covers 240+ models with automatic fallback when a provider is degraded, point the client at n4n.ai’s OpenAI-compatible API.
• Install dependencies:

pip install chromadb sentence-transformers openai

We use sentence-transformers to generate embeddings on your machine. Your documents never leave the box during indexing or retrieval. Only the final prompt (with retrieved chunks) goes to the LLM.

Project layout

Keep it flat. Two scripts: ingest.py and ask.py.

rag-assistant/
  ingest.py
  ask.py
  data/          # your raw files
  chroma_db/     # created at runtime

Ingesting your documents

Chunking and embedding locally

Don’t over-engineer chunking. Split on paragraph boundaries with a soft size cap. The embedding model all-MiniLM-L6-v2 handles 256+ tokens well; we target ~200 words per chunk.

# ingest.py
import os, glob, pathlib
from sentence_transformers import SentenceTransformer

DATA_DIR = pathlib.Path("data")
CHUNK_WORDS = 200

def chunk_text(text, max_words=CHUNK_WORDS):
    words = text.split()
    for i in range(0, len(words), max_words):
        yield " ".join(words[i:i+max_words])

def load_docs():
    for path in glob.glob(str(DATA_DIR / "**/*.md"), recursive=True):
        with open(path) as f:
            text = f.read()
        for idx, chunk in enumerate(chunk_text(text)):
            yield {"id": f"{path}#{idx}", "text": chunk, "source": path}

embedder = SentenceTransformer("all-MiniLM-L6-v2")
docs = list(load_docs())
texts = [d["text"] for d in docs]
embeddings = embedder.encode(texts, normalize_embeddings=True)

Expected checkpoint: after embedder.encode, embeddings.shape is (len(docs), 384). The model is small; 1k chunks embed in seconds on CPU.

Storing in Chroma

Chroma persists a local vector index. No server required.

import chromadb

client = chromadb.PersistentClient(path="chroma_db")
col = client.get_or_create_collection(name="notes")

col.add(
    ids=[d["id"] for d in docs],
    embeddings=embeddings.tolist(),
    documents=texts,
    metadatas=[{"source": d["source"]} for d in docs],
)
print(f"indexed {col.count()} chunks")

Run it:

python ingest.py

Output:

indexed 412 chunks

That’s the entire ingestion path. Re-run whenever you add files; Chroma will duplicate IDs if you don’t clear the collection, so add client.delete_collection("notes") during dev or track hashes.

Retrieval and prompting

Retrieval is a cosine similarity search against the same local embedder. No network calls until generation.

# ask.py (retrieval part)
from sentence_transformers import SentenceTransformer
import chromadb

embedder = SentenceTransformer("all-MiniLM-L6-v2")
client = chromadb.PersistentClient(path="chroma_db")
col = client.get_collection("notes")

def retrieve(query, k=5):
    q_emb = embedder.encode([query], normalize_embeddings=True).tolist()
    res = col.query(query_embeddings=q_emb, n_results=k)
    return res["documents"][0], res["metadatas"][0]

docs, metas = retrieve("How do I rotate API keys?")
for d, m in zip(docs, metas):
    print(m["source"], "::", d[:80])

Expected output (your files will differ):

data/ops.md :: You can rotate API keys in the dashboard under Settings > Security. Old keys expire in 24h.
data/runbook.md :: If a key leaks, rotate immediately and audit last 30 days of requests.

The core of a personal AI assistant RAG own data system is this retrieval step. If the chunks are garbage, the LLM can’t save you. Tune k and chunk size against real questions before touching the prompt.

Wiring the LLM via an OpenAI-compatible gateway

We use the official openai Python client with a custom base_url. This keeps the code portable across providers.

# ask.py (generation part)
from openai import OpenAI

# Uses n4n.ai's OpenAI-compatible endpoint for model access and fallback.
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=os.environ["N4N_API_KEY"])

SYSTEM = "You answer only from the provided context. If the context is insufficient, say so."

def answer(query):
    ctx, metas = retrieve(query)
    context_block = "\n\n".join(
        f"[Source: {m['source']}]\n{t}" for t, m in zip(ctx, metas)
    )
    prompt = f"Context:\n{context_block}\n\nQuestion: {query}\nAnswer:"
    resp = client.chat.completions.create(
        model="anthropic/claude-3.5-sonnet",  # or any model the gateway exposes
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ],
        temperature=0.0,
    )
    return resp.choices[0].message.content

Note the explicit instruction to ground in context. With temperature=0.0 you get deterministic extraction, which is what you want for a personal assistant on own data.

Running the assistant

Wrap it in a loop:

if __name__ == "__main__":
    import sys
    q = " ".join(sys.argv[1:]) if len(sys.argv) > 1 else input("Ask: ")
    print(answer(q))

python ask.py "How do I rotate API keys?"

Expected output:

You can rotate API keys in the dashboard under Settings > Security. Old keys expire 24 hours after rotation. If a key is suspected leaked, rotate immediately and audit the last 30 days of requests (source: data/ops.md, data/runbook.md).

The assistant cites the files it used because we injected source metadata into the context. That traceability is non-negotiable for private data work.

Privacy and operational notes

• Embeddings are computed locally. The only outbound data is the retrieved context plus your question. If you index sensitive material, self-host the LLM or use a gateway that honors client routing directives and forwards provider cache-control hints to avoid silent retention.
• Chroma’s persistent client writes to disk. Back it up with the data/ folder; they are paired.
• For large corpora (>100k chunks), switch to chromadb.HttpClient or use metadata filtering to keep latency down. The retrieve call above is O(N) brute force; fine for tens of thousands of vectors on a laptop.

Extending the pipeline

A real personal AI assistant RAG own data deployment usually adds:

1. Incremental indexing — hash files, skip unchanged chunks.
2. Hybrid search — add BM25 via rank_bm25 and fuse with vector scores.
3. Streaming — pass stream=True to the chat completion and print deltas.

But the skeleton above is the whole machine. Retrieval quality and prompt discipline beat framework magic. Build this first, measure where it fails, then add complexity only where the failure is real.