Building a local RAG app with Mistral 7B and Ollama

Running retrieval-augmented generation on your own hardware removes API latency, cuts recurring costs, and keeps sensitive documents on the box. This tutorial builds a local rag app mistral 7b ollama that ingests plain-text files, embeds them with a local embedding model, and answers questions using Mistral 7B served by Ollama. Everything runs offline, no cloud keys required.

Step 1: Install Ollama and pull the models

Start by installing Ollama for your platform (macOS, Linux, or Windows WSL). The binary bundles a model runner that handles quantization and inference. After install, pull the two models we need:

ollama pull mistral:7b
ollama pull nomic-embed-text

mistral:7b is the 7-billion-parameter Mistral model in 4-bit quantization, small enough to run on a laptop GPU or even CPU. nomic-embed-text is a local embedding model that produces 768-dimensional vectors, keeping the embedding step fully offline. Verify they are present:

ollama list

You should see both tags. If you later swap Mistral for a different local LLM, the rest of the pipeline stays identical as long as the Ollama API contract holds.

Step 2: Set up the Python environment

Create an isolated environment and install the only two dependencies required: the official ollama Python client and chromadb for vector storage.

python -m venv venv
source venv/bin/activate
pip install ollama chromadb

We avoid heavy RAG frameworks. The Ollama client talks to the local daemon over HTTP; Chroma runs in-process for development. For production you’d point Chroma at a persistent backend, but the local rag app mistral 7b ollama works fine with an embedded store during prototyping.

Step 3: Ingest and chunk documents

Retrieval quality depends on chunk size. Too large, and embeddings blur topics; too small, and context gets fragmented. A 500-character window with 50-character overlap is a sane default for prose.

Create ingest.py:

import ollama
import chromadb
from chromadb.config import Settings

client = chromadb.Client(Settings(anonymized_telemetry=False))
collection = client.create_collection("docs")

def chunk_text(text, size=500, overlap=50):
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start+size])
        start += size - overlap
    return chunks

with open("sample.txt") as f:
    text = f.read()

chunks = chunk_text(text)
for i, chunk in enumerate(chunks):
    emb = ollama.embeddings(model="nomic-embed-text", prompt=chunk)["embedding"]
    collection.add(ids=[str(i)], embeddings=[emb], documents=[chunk])

print(f"Ingested {len(chunks)} chunks")

Run it with python ingest.py. Success is indicated by the printed count and no exceptions. The nomic-embed-text model must match between ingest and query; mixing embedding models breaks cosine similarity.

Chunking strategy note

For code or structured data, consider splitting on delimiters (e.g., Markdown headers) instead of fixed windows. The local rag app mistral 7b ollama pipeline treats each chunk as a standalone retrieval unit, so preserve semantic boundaries where possible. If your source is a set of JSON records, embed each record as one chunk to avoid cross-entity contamination.

Step 4: Retrieve relevant context

Query-time embedding must use the same model as ingestion. Build a retrieve function that returns the top-k chunks:

def retrieve(query, k=3):
    emb = ollama.embeddings(model="nomic-embed-text", prompt=query)["embedding"]
    results = collection.query(query_embeddings=[emb], n_results=k)
    return results["documents"][0]

Chroma returns documents in order of similarity. For a sample.txt of 2,000 words, three chunks usually suffice to ground Mistral’s answer. If you need stricter grounding, raise k to 5 and truncate to fit the model context.

Step 5: Generate answers with Mistral 7B

Ollama’s generate endpoint accepts a prompt and sampling options. Constrain the model to use only provided context to reduce hallucination:

def ask(query):
    context = "\n\n".join(retrieve(query))
    prompt = f"""Answer the question using only the context.

Context:
{context}

Question: {query}
Answer:"""
    resp = ollama.generate(
        model="mistral:7b",
        prompt=prompt,
        options={"temperature": 0.0, "num_ctx": 2048}
    )
    return resp["response"]

if __name__ == "__main__":
    print(ask("What is the refund policy described in the document?"))

Setting temperature to 0 makes outputs deterministic. num_ctx expands the context window to fit chunks plus question. On a 16GB machine, Mistral 7B Q4 consumes about 4–5 GB VRAM; if you’re on CPU-only, expect 10–20 tokens/sec. The local rag app mistral 7b ollama benefits from a system prompt if you want stricter tone control—pass system="You are a precise document analyst." in the generate call.

Step 6: Run end-to-end and verify

Execute the full flow:

python ingest.py
python query.py

Verification: feed a question whose answer is verbatim in sample.txt. If the printed answer matches the source text and cites no external facts, the local rag app mistral 7b ollama works. A failure mode is empty retrieval—check that ollama embeddings returns non-zero vectors and that Chroma collection name matches.

Add a quick assertion script to catch regressions:

def test_retrieval():
    hits = retrieve("refund policy")
    assert any("refund" in h.lower() for h in hits)

Run it under pytest after ingestion. If the assertion passes, your embedding and chunking logic is wired correctly.

Step 7: Persist and expose as a service

For repeated use, switch to a persistent Chroma client and wrap ask in a tiny HTTP server. Replace the client line in ingest.py with:

client = chromadb.PersistentClient(path="./chroma_store")

Then a FastAPI endpoint:

from fastapi import FastAPI
app = FastAPI()

@app.get("/ask")
def ask_endpoint(q: str):
    return {"answer": ask(q)}

Run with uvicorn query:app --reload. The model stays loaded in the Ollama daemon, so subsequent requests skip model load overhead.

Operational notes

• Ollama daemon caches the model in RAM; first call after pull is slower.
• If you run multiple local models, ollama ps shows active ones.
• The same pattern scales to Llama 3, DeepSeek, or Qwen by changing the model string—the local rag app mistral 7b ollama is a template, not a lock-in.
• Chroma’s persistent client writes to disk; back it up if the ingested corpus is large.

Wrapping up

You now have a private RAG loop: embed locally, store in Chroma, retrieve, and generate with Mistral 7B. Extend it by adding PDF loaders (pypdf), metadata filtering in Chroma, or hybrid search with BM25. The core contract—Ollama for inference and embeddings, a vector store for context—remains stable as you swap models or add document types.