Integrate Mistral Small 3.1 into a LlamaIndex RAG pipeline

Mistral Small 3.1 delivers strong multilingual reasoning and a 128k context window at a fraction of the cost of larger models, making it an excellent choice for RAG workloads. This tutorial walks through a complete mistral small llamaindex rag integration from environment setup through a queryable pipeline, with checkpoints you can verify at each step.

Prerequisites

Before starting, ensure you have:

• Python 3.10 or newer
• An API key for a provider serving Mistral Small 3.1 (Together AI, Fireworks AI, or a unified gateway like n4n.ai)
• Basic familiarity with LlamaIndex concepts: documents, nodes, indexes, and query engines

Install the required packages:

pip install llama-index llama-index-llms-openai-like llama-index-embeddings-huggingface pypdf python-dotenv

We use llama-index-llms-openai-like because Mistral Small 3.1 is served behind an OpenAI-compatible API. The Hugging Face embedding model runs locally, keeping embedding costs at zero.

Configure the environment

Create a .env file in your project root:

# .env
MISTRAL_API_KEY=your_api_key_here
MISTRAL_BASE_URL=https://api.together.xyz/v1  # or your provider's base URL
EMBEDDING_MODEL=BAAI/bge-small-en-v1.5

If you’re using a unified gateway that aggregates multiple providers, the base URL and key will differ — adjust accordingly. The rest of the code remains unchanged.

Load the configuration in a config.py module:

# config.py
import os
from dotenv import load_dotenv

load_dotenv()

MISTRAL_API_KEY = os.getenv("MISTRAL_API_KEY")
MISTRAL_BASE_URL = os.getenv("MISTRAL_BASE_URL", "https://api.together.xyz/v1")
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "BAAI/bge-small-en-v1.5")

if not MISTRAL_API_KEY:
    raise ValueError("MISTRAL_API_KEY not set in environment")

Initialize the LLM and embedding model

Create models.py to encapsulate model initialization. This keeps your pipeline code clean and makes swapping providers trivial.

# models.py
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from config import MISTRAL_API_KEY, MISTRAL_BASE_URL, EMBEDDING_MODEL

def get_llm() -> OpenAILike:
    """Return a configured Mistral Small 3.1 client."""
    return OpenAILike(
        model="mistralai/Mistral-Small-3.1-24B-Instruct-2503",
        api_key=MISTRAL_API_KEY,
        api_base=MISTRAL_BASE_URL,
        temperature=0.1,
        max_tokens=4096,
        is_chat_model=True,
        timeout=60,
    )

def get_embed_model() -> HuggingFaceEmbedding:
    """Return a local embedding model."""
    return HuggingFaceEmbedding(model_name=EMBEDDING_MODEL)

Checkpoint: Run a quick sanity check in a REPL:

from models import get_llm, get_embed_model

llm = get_llm()
resp = llm.complete("Say 'ready' if you can hear me.")
print(resp.text)  # Should print: ready

Prepare sample data

For this tutorial, we’ll ingest a few PDFs. Create a data/ directory and drop in 2–3 PDF files (technical docs, research papers, or any text-heavy PDFs). If you don’t have any handy, generate a couple of dummy PDFs:

# generate_dummy_data.py
from fpdf import FPDF
import os

os.makedirs("data", exist_ok=True)

docs = {
    "mistral_overview.pdf": (
        "Mistral Small 3.1 Overview\n\n"
        "Mistral Small 3.1 is a 24B parameter model released in March 2025. "
        "It supports 128k context length and excels at multilingual tasks, "
        "coding, and reasoning. The model is available under Apache 2.0 license."
    ),
    "rag_best_practices.pdf": (
        "RAG Best Practices\n\n"
        "Effective retrieval-augmented generation requires: "
        "1) Chunking strategy matched to query type. "
        "2) Hybrid search combining dense and sparse retrieval. "
        "3) Reranking to improve precision. "
        "4) Citation-aware generation to reduce hallucination."
    ),
    "llamaindex_architecture.pdf": (
        "LlamaIndex Architecture\n\n"
        "LlamaIndex provides a modular framework for building LLM applications. "
        "Core modules include: Data Connectors, Indexes, Retrievers, Query Engines, "
        "and Agents. Each module can be customized or replaced independently."
    ),
}

for filename, content in docs.items():
    pdf = FPDF()
    pdf.add_page()
    pdf.set_font("Helvetica", size=12)
    pdf.multi_cell(0, 6, content)
    pdf.output(f"data/{filename}")

print("Generated 3 dummy PDFs in data/")

Run it:

pip install fpdf2
python generate_dummy_data.py

Build the ingestion pipeline

The ingestion pipeline handles loading, chunking, embedding, and persisting documents. Create ingest.py:

# ingest.py
from pathlib import Path
from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    StorageContext,
    Settings,
)
from llama_index.core.node_parser import SentenceSplitter
from models import get_llm, get_embed_model

DATA_DIR = Path("data")
PERSIST_DIR = Path("storage")

def build_index() -> VectorStoreIndex:
    # Configure global settings
    Settings.llm = get_llm()
    Settings.embed_model = get_embed_model()
    Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=64)

    # Load documents
    documents = SimpleDirectoryReader(DATA_DIR).load_data()
    print(f"Loaded {len(documents)} documents")

    # Build index
    index = VectorStoreIndex.from_documents(documents, show_progress=True)

    # Persist to disk
    PERSIST_DIR.mkdir(exist_ok=True)
    index.storage_context.persist(persist_dir=PERSIST_DIR)
    print(f"Index persisted to {PERSIST_DIR}")

    return index

if __name__ == "__main__":
    build_index()

Run the ingestion:

python ingest.py

Expected output:

Loaded 3 documents
Index persisted to storage

The storage/ directory now contains your vector index (default: in-memory vectors serialized to JSON). For production, swap the vector store for Pinecone, Weaviate, or Qdrant — LlamaIndex makes this a one-line change.

Create the query engine

With the index persisted, build a query engine that retrieves relevant chunks and synthesizes answers with citations. Create query.py:

# query.py
from pathlib import Path
from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
    get_response_synthesizer,
)
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SimilarityPostprocessor
from models import get_llm, get_embed_model

PERSIST_DIR = Path("storage")

def get_query_engine() -> RetrieverQueryEngine:
    Settings.llm = get_llm()
    Settings.embed_model = get_embed_model()

    # Load persisted index
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = VectorStoreIndex.from_vector_store(
        storage_context.vector_store,
        storage_context=storage_context,
    )

    # Configure retriever: top-5 with similarity threshold
    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=5,
    )

    # Post-process: filter low-similarity nodes
    postprocessor = SimilarityPostprocessor(similarity_cutoff=0.7)

    # Response synthesizer with citation support
    response_synthesizer = get_response_synthesizer(
        response_mode="compact",
        use_async=False,
    )

    query_engine = RetrieverQueryEngine(
        retriever=retriever,
        response_synthesizer=response_synthesizer,
        node_postprocessors=[postprocessor],
    )

    return query_engine

def main():
    query_engine = get_query_engine()

    questions = [
        "What is the context length of Mistral Small 3.1?",
        "What are the key components of LlamaIndex architecture?",
        "List three RAG best practices mentioned in the documents.",
    ]

    for q in questions:
        print(f"\n{'='*60}")
        print(f"Q: {q}")
        print(f"{'='*60}")
        response = query_engine.query(q)
        print(f"A: {response.response}")
        print("\nSources:")
        for i, node in enumerate(response.source_nodes):
            print(f"  [{i+1}] {node.metadata.get('file_name', 'unknown')} "
                  f"(score: {node.score:.3f})")
            print(f"      {node.text[:120]}...")

if __name__ == "__main__":
    main()

Run it:

python query.py

Expected output (scores will vary slightly):

============================================================
Q: What is the context length of Mistral Small 3.1?
============================================================
A: Mistral Small 3.1 supports a 128k context length.

Sources:
  [1] mistral_overview.pdf (score: 0.842)
      Mistral Small 3.1 is a 24B parameter model released in March 2025. 
      It supports 128k context length and excels at multilingual tasks...

============================================================
Q: What are the key components of LlamaIndex architecture?
============================================================
A: The key components of LlamaIndex architecture are Data Connectors, 
Indexes, Retrievers, Query Engines, and Agents.

Sources:
  [1] llamaindex_architecture.pdf (score: 0.891)
      LlamaIndex provides a modular framework for building LLM applications. 
      Core modules include: Data Connectors, Indexes, Retrievers, Query Engines...

============================================================
Q: List three RAG best practices mentioned in the documents.
============================================================
A: 1) Chunking strategy matched to query type. 
2) Hybrid search combining dense and sparse retrieval. 
3) Reranking to improve precision.

Sources:
  [1] rag_best_practices.pdf (score: 0.867)
      Effective retrieval-augmented generation requires: 
      1) Chunking strategy matched to query type. 
      2) Hybrid search combining dense and sparse retrieval...

The pipeline works: retrieval finds the right chunks, the similarity cutoff filters noise, and the response synthesizer produces grounded answers with citations.

Add streaming for better UX

For interactive applications, streaming tokens as they arrive improves perceived latency. Update query.py to support streaming:

# query_streaming.py
from pathlib import Path
from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
    get_response_synthesizer,
)
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SimilarityPostprocessor
from models import get_llm, get_embed_model

PERSIST_DIR = Path("storage")

def get_streaming_query_engine() -> RetrieverQueryEngine:
    Settings.llm = get_llm()
    Settings.embed_model = get_embed_model()

    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = VectorStoreIndex.from_vector_store(
        storage_context.vector_store,
        storage_context=storage_context,
    )

    retriever = VectorIndexRetriever(index=index, similarity_top_k=5)
    postprocessor = SimilarityPostprocessor(similarity_cutoff=0.7)

    # Streaming response synthesizer
    response_synthesizer = get_response_synthesizer(
        response_mode="compact",
        use_async=False,
        streaming=True,
    )

    return RetrieverQueryEngine(
        retriever=retriever,
        response_synthesizer=response_synthesizer,
        node_postprocessors=[postprocessor],
    )

def main():
    query_engine = get_streaming_query_engine()
    question = "Summarize the key advantages of Mistral Small 3.1 for RAG applications."

    print(f"Q: {question}\n")
    print("A: ", end="", flush=True)

    streaming_response = query_engine.query(question)
    for token in streaming_response.response_gen:
        print(token, end="", flush=True)

    print("\n\nSources:")
    for i, node in enumerate(streaming_response.source_nodes):
        print(f"  [{i+1}] {node.metadata.get('file_name', 'unknown')} "
              f"(score: {node.score:.3f})")

if __name__ == "__main__":
    main()

Run it:

python query_streaming.py

Expected output (tokens appear incrementally):

Q: Summarize the key advantages of Mistral Small 3.1 for RAG applications.

A: Mistral Small 3.1 offers several key advantages for RAG applications: 
it has a 128k context window allowing large document ingestion, strong 
multilingual capabilities for diverse corpora, solid reasoning for 
synthesis tasks, and is available under Apache 2.0 license enabling 
commercial deployment.

Sources:
  [1] mistral_overview.pdf (score: 0.842)
      Mistral Small 3.1 is a 24B parameter model released in March 2025. 
      It supports 128k context length and excels at multilingual tasks...

Evaluate retrieval quality

Before deploying, measure retrieval precision. Create evaluate.py:

# evaluate.py
from pathlib import Path
from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
)
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.schema import QueryBundle
from models import get_llm, get_embed_model

PERSIST_DIR = Path("storage")

# Ground truth: (question, expected_source_file)
TEST_CASES = [
    ("What is the context length of Mistral Small 3.1?", "mistral_overview.pdf"),
    ("What are the core modules of LlamaIndex?", "llamaindex_architecture.pdf"),
    ("What does hybrid search combine?", "rag_best_practices.pdf"),
]

def evaluate_retrieval(k=5, cutoff=0.7):
    Settings.llm = get_llm()
    Settings.embed_model = get_embed_model()

    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = VectorStoreIndex.from_vector_store(
        storage_context.vector_store,
        storage_context=storage_context,
    )

    retriever = VectorIndexRetriever(index=index, similarity_top_k=k)

    hits = 0
    for question, expected_file in TEST_CASES:
        nodes = retriever.retrieve(QueryBundle(query_str=question))
        # Filter by cutoff
        nodes = [n for n in nodes if n.score >= cutoff]
        
        found = any(expected_file in n.metadata.get("file_name", "") for n in nodes)
        status = "✓" if found else "✗"
        print(f"{status} {question}")
        if not found:
            print(f"   Expected: {expected_file}")
            print(f"   Got: {[n.metadata.get('file_name') for n in nodes[:3]]}")
        if found:
            hits += 1

    print(f"\nRecall@{k} (cutoff={cutoff}): {hits}/{len(TEST_CASES)} = {hits/len(TEST_CASES):.1%}")

if __name__ == "__main__":
    evaluate_retrieval()

Run it:

python evaluate.py

Expected output:

✓ What is the context length of Mistral Small 3.1?
✓ What are the core modules of LlamaIndex?
✓ What does hybrid search combine?

Recall@5 (cutoff=0.7): 3/3 = 100.0%

With three test cases this is a sanity check, not a rigorous eval. For production, build a larger labeled dataset and track recall@k, MRR, and nDCG over time.

Production hardening

Three changes move this from tutorial to production-ready:

1. Hybrid search

Dense vectors miss exact matches (IDs, error codes, proper nouns). Add BM25:

# hybrid_retriever.py
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core import VectorStoreIndex

def get_hybrid_retriever(index: VectorStoreIndex, similarity_top_k: int = 5):
    vector_retriever = index.as_retriever(similarity_top_k=similarity_top_k)
    bm25_retriever = BM25Retriever.from_defaults(
        index=index, similarity_top_k=similarity_top_k
    )
    return QueryFusionRetriever(
        [vector_retriever, bm25_retriever],
        similarity_top_k=similarity_top_k,
        num_queries=1,  # set >1 for query rewriting
        mode="reciprocal_rerank",
        use_async=True,
    )

2. Reranking

A cross-encoder reranker improves precision significantly:

# reranker.py
from llama_index.postprocessor.cohere_rerank import CohereRerank
from llama_index.postprocessor.llm_rerank import LLMRerank

# Option A: Cohere (fast, hosted)
cohere_rerank = CohereRerank(top_n=3, model="rerank-v3.5")

# Option B: Local LLM reranker (no external API)
llm_rerank = LLMRerank(
    choice_batch_size=5,
    top_n=3,
    llm=Settings.llm,
)

Add either to node_postprocessors in your query engine.

3. Observability

Instrument with OpenTelemetry or LlamaIndex’s built-in callbacks:

# observability.py
from llama_index.core import set_global_handler
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
import tiktoken

token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-4").encode
)
set_global_handler("simple")  # or "wandb", "arize", "langfuse"
Settings.callback_manager = CallbackManager([token_counter])

After running queries, token_counter.total_llm_token_count gives you exact usage for cost tracking.

Common pitfalls

Next steps

You now have a working mistral small llamaindex rag integration with:

• Document ingestion and persistence
• Configurable retrieval with similarity filtering
• Citation-backed generation (streaming and non-streaming)
• Evaluation harness
• Clear path to hybrid search, reranking, and observability

From here, consider:

1. Swap the vector store — pip install llama-index-vector-stores-qdrant and change two lines
2. Add a query router — route simple lookups to BM25, complex reasoning to dense retrieval
3. Implement query rewriting — expand ambiguous queries before retrieval
4. Build an agent — give the LLM tools to iteratively retrieve, read, and synthesize

The mistral small llamaindex rag integration pattern scales from prototype to production with the same core abstractions. Start simple, measure, then add complexity only where metrics demand it.