LlamaIndex plus Ollama: indexing docs with DeepSeek-V3

You want to index and query documents locally without sending data to external APIs, but you still want a frontier-class model for synthesis. This tutorial shows how to wire LlamaIndex with Ollama for embeddings and DeepSeek-V3 via an OpenAI-compatible endpoint for generation. You’ll end up with a runnable pipeline that ingests a directory of files, builds a vector index, and answers questions with citations.

Prerequisites

• Python 3.10+
• Ollama installed and running (ollama serve)
• An OpenAI-compatible endpoint for DeepSeek-V3 (OpenRouter, n4n.ai, or self-hosted vLLM)
• A directory of documents to index (Markdown, PDF, text)

Install the dependencies:

pip install llama-index llama-index-llms-ollama llama-index-embeddings-ollama \
  llama-index-readers-file pypdf python-dotenv

Pull the embedding model in Ollama. We’ll use nomic-embed-text — 137M parameters, fast, and strong retrieval quality:

ollama pull nomic-embed-text

Verify it’s available:

ollama list
# NAME                ID        SIZE    MODIFIED
# nomic-embed-text    latest    274 MB  2 minutes ago

Create a .env file for your DeepSeek-V3 endpoint:

# .env
DEEPSEEK_API_KEY=sk-or-v1-xxxxxxxxxxxxx
DEEPSEEK_BASE_URL=https://openrouter.ai/api/v1
# Or if using n4n.ai:
# DEEPSEEK_BASE_URL=https://api.n4n.ai/v1

Project structure

rag-pipeline/
├── .env
├── data/
│   ├── doc1.md
│   ├── doc2.pdf
│   └── notes.txt
├── ingest.py
├── query.py
└── requirements.txt

Put a few files in data/ — any mix of Markdown, PDF, or plain text works.

Step 1: Configure LlamaIndex settings

Create settings.py to centralize configuration. This avoids scattering model names and parameters across scripts.

# settings.py
import os
from dotenv import load_dotenv
from llama_index.core import Settings
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.openai_like import OpenAILike

load_dotenv()

# Local embeddings via Ollama
Settings.embed_model = OllamaEmbedding(
    model_name="nomic-embed-text",
    base_url="http://localhost:11434",
    # Optional: tune for throughput
    # request_timeout=120.0,
)

# DeepSeek-V3 via OpenAI-compatible endpoint
Settings.llm = OpenAILike(
    model="deepseek/deepseek-v3",  # OpenRouter model slug
    api_key=os.getenv("DEEPSEEK_API_KEY"),
    api_base=os.getenv("DEEPSEEK_BASE_URL"),
    is_chat_model=True,
    temperature=0.1,
    max_tokens=4096,
    # Context window for DeepSeek-V3
    context_window=128000,
)

# Chunking defaults — adjust for your corpus
Settings.chunk_size = 1024
Settings.chunk_overlap = 128

Run a quick sanity check:

python -c "from settings import Settings; print('Embed model:', Settings.embed_model.model_name); print('LLM:', Settings.llm.model)"

Expected output:

Embed model: nomic-embed-text
LLM: deepseek/deepseek-v3

Step 2: Build the ingestion script

ingest.py loads documents, splits them, embeds chunks, and persists a vector store to disk. We use LlamaIndex’s SimpleDirectoryReader and the default in-memory vector store backed by faiss (install faiss-cpu if you want persistence; otherwise it pickles the index).

# ingest.py
import sys
from pathlib import Path
from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    StorageContext,
)
from llama_index.core.node_parser import SentenceSplitter
from settings import Settings

DATA_DIR = Path("data")
PERSIST_DIR = Path("storage")

def main():
    if not DATA_DIR.exists():
        print(f"Data directory {DATA_DIR} does not exist")
        sys.exit(1)

    print(f"Loading documents from {DATA_DIR}...")
    documents = SimpleDirectoryReader(
        input_dir=str(DATA_DIR),
        recursive=True,
        required_exts=[".md", ".pdf", ".txt", ".html"],
    ).load_data()

    print(f"Loaded {len(documents)} documents")
    for doc in documents:
        print(f"  - {doc.metadata.get('file_name', 'unknown')}: {len(doc.text)} chars")

    # Explicit node parser gives you control over chunking
    parser = SentenceSplitter(
        chunk_size=Settings.chunk_size,
        chunk_overlap=Settings.chunk_overlap,
    )
    nodes = parser.get_nodes_from_documents(documents, show_progress=True)
    print(f"Created {len(nodes)} nodes")

    # Build index
    print("Building vector index...")
    index = VectorStoreIndex(nodes, show_progress=True)

    # Persist to disk
    PERSIST_DIR.mkdir(exist_ok=True)
    index.storage_context.persist(persist_dir=str(PERSIST_DIR))
    print(f"Index persisted to {PERSIST_DIR}")

if __name__ == "__main__":
    main()

Run it:

python ingest.py

Expected output (varies by corpus):

Loading documents from data...
Loaded 3 documents
  - doc1.md: 12450 chars
  - doc2.pdf: 8921 chars
  - notes.txt: 3102 chars
Created 47 nodes
Building vector index...
Index persisted to storage

The storage/ directory now contains default__vector_store.json, docstore.json, and index_store.json.

Step 3: Query the index

query.py loads the persisted index, configures a retriever, and runs a query engine with citation support.

# query.py
import sys
from pathlib import Path
from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    load_index_from_storage,
)
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.response_synthesizers import CompactAndRefine
from settings import Settings

PERSIST_DIR = Path("storage")

def build_query_engine():
    if not PERSIST_DIR.exists():
        print(f"Index not found at {PERSIST_DIR}. Run ingest.py first.")
        sys.exit(1)

    print("Loading index from storage...")
    storage_context = StorageContext.from_defaults(persist_dir=str(PERSIST_DIR))
    index = load_index_from_storage(storage_context)

    # Configure retriever: top-k with similarity threshold
    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=6,
        # Optional: filter by metadata
        # filters=MetadataFilters(...),
    )

    # CompactAndRefine fits more context per LLM call than TreeSummarize
    synthesizer = CompactAndRefine(
        llm=Settings.llm,
        verbose=True,
    )

    query_engine = RetrieverQueryEngine(
        retriever=retriever,
        response_synthesizer=synthesizer,
    )
    return query_engine

def main():
    if len(sys.argv) < 2:
        print("Usage: python query.py \"your question here\"")
        sys.exit(1)

    question = " ".join(sys.argv[1:])
    print(f"\nQuestion: {question}\n")

    engine = build_query_engine()
    response = engine.query(question)

    print("=" * 60)
    print("ANSWER")
    print("=" * 60)
    print(response.response)
    print()

    if response.source_nodes:
        print("=" * 60)
        print("SOURCES")
        print("=" * 60)
        for i, node in enumerate(response.source_nodes, 1):
            meta = node.metadata
            file_name = meta.get("file_name", "unknown")
            page = meta.get("page_label", meta.get("page", "?"))
            score = node.score
            print(f"\n[{i}] {file_name} (page {page}) — score: {score:.4f}")
            print(f"    {node.text[:200]}...")

if __name__ == "__main__":
    main()

Test it:

python query.py "What are the key differences between the approaches described in the documents?"

Expected output shape:

Question: What are the key differences between the approaches described in the documents?

Loading index from storage...

ANSWER
============================================================
Based on the provided documents, there are three key differences...

============================================================
SOURCES
============================================================

[1] doc1.md (page 1) — score: 0.8234
    The first approach uses a centralized coordinator...

[2] doc2.pdf (page 3) — score: 0.7912
    In contrast, the distributed model eliminates...

[3] notes.txt (page ?) — score: 0.7105
    Hybrid variants attempt to combine...

Step 4: Streaming responses

For interactive use, stream tokens as they arrive. Replace the query call in query.py:

# Streaming variant
response = engine.query(question)

print("ANSWER")
print("=" * 60)
for token in response.response_gen:
    print(token, end="", flush=True)
print("\n")

The response_gen attribute exists on StreamingResponse — enable it by passing streaming=True to the query engine constructor:

query_engine = RetrieverQueryEngine(
    retriever=retriever,
    response_synthesizer=synthesizer,
    streaming=True,
)

Step 5: Hybrid retrieval (BM25 + vector)

Pure vector search misses exact keyword matches. Add a BM25 retriever and fuse results with QueryFusionRetriever:

# hybrid_query.py
from llama_index.core.retrievers import QueryFusionRetriever, BM25Retriever
from llama_index.core import VectorStoreIndex, StorageContext, load_index_from_storage
from settings import Settings

def build_hybrid_engine(persist_dir="storage"):
    storage_context = StorageContext.from_defaults(persist_dir=persist_dir)
    index = load_index_from_storage(storage_context)

    vector_retriever = VectorIndexRetriever(index=index, similarity_top_k=6)
    bm25_retriever = BM25Retriever.from_defaults(
        index=index,
        similarity_top_k=6,
    )

    hybrid = QueryFusionRetriever(
        retrievers=[vector_retriever, bm25_retriever],
        similarity_top_k=6,
        num_queries=1,  # set >1 for query rewriting
        mode="reciprocal_rerank",
        use_async=True,
        verbose=True,
    )

    return RetrieverQueryEngine.from_args(
        retriever=hybrid,
        llm=Settings.llm,
        streaming=True,
    )

Run with the same CLI. The fusion retriever often surfaces documents that vector-only misses — especially for proper nouns, error codes, or acronyms.

Step 6: Persistent vector store with Chroma

The default in-memory store works for prototypes. For production, swap in Chroma:

pip install chromadb llama-index-vector-stores-chroma

# ingest_chroma.py
import chromadb
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
from settings import Settings

chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("docs")

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex(nodes, storage_context=storage_context, show_progress=True)
# No need to call persist() — Chroma writes automatically

Querying uses the same load_index_from_storage pattern with the Chroma-backed storage context.

Tuning knobs worth knowing

Common failure modes

Ollama connection refused — Ensure ollama serve is running and base_url matches (default http://localhost:11434). Check curl http://localhost:11434/api/tags.

DeepSeek-V3 returns 401/404 — Verify DEEPSEEK_API_KEY and DEEPSEEK_BASE_URL. The model slug must match the provider’s catalog (deepseek/deepseek-v3 on OpenRouter, deepseek-v3 on n4n.ai).

Out of memory on embedding — nomic-embed-text fits in 2–4 GB VRAM. If OOM, try all-minilm:33m (smaller, lower quality) or batch embedding: Settings.embed_model.embed_batch_size = 16.

PDF extraction yields garbage — pypdf struggles with scanned PDFs. Add pdfplumber or marker for complex layouts:

pip install pdfplumber

from llama_index.readers.file import PDFReader
reader = PDFReader(pdf_reader="pdfplumber")
documents = reader.load_data(file=Path("data/doc2.pdf"))

What this gives you

• Local embeddings: Document content never leaves your machine for vectorization.
• Frontier generation: DeepSeek-V3 handles synthesis via an OpenAI-compatible endpoint — swap providers without code changes.
• Citations: Every answer includes source nodes with scores and metadata.
• Extensible: Add rerankers (Cohere, Jina, local bge-reranker), hybrid search, or agentic workflows on top.

The same pattern scales: swap nomic-embed-text for bge-m3 (multilingual), point the LLM at a local llama3.1:70b via Ollama for fully air-gapped operation, or route through a gateway that handles fallback and metering across providers. The LlamaIndex abstractions stay the same.