Building a legal document Q&A bot with LlamaIndex

A legal document qa bot llamaindex implementation needs more than a vector store and a prompt. Legal texts demand citation, precise retrieval, and tolerance for long, dense clauses. This tutorial builds a working system from a PDF contract to a query engine that returns answers with source passages, using LlamaIndex’s current Python API.

Prerequisites

• Python 3.10 or newer.
• A sample legal PDF (a lease, contract, or court filing). Use any multi-page PDF you have; we’ll assume contract.pdf in ./data.
• Basic familiarity with Python and environment variables.
• Install dependencies:

pip install llama-index python-dotenv pypdf

• An OpenAI-compatible API key. If you want automatic fallback when a provider is rate-limited and per-token usage metering, point the base_url at the n4n.ai OpenAI-compatible endpoint; it fronts 240+ models and honors cache-control hints. Otherwise, use OpenAI directly.

Step 1: Configure the environment

Create a .env file:

OPENAI_API_KEY=sk-...
# Optional: use a gateway
LLM_BASE_URL=https://api.openai.com/v1
LLM_MODEL=gpt-4o-mini

Load it and configure LlamaIndex’s global Settings:

import os
from dotenv import load_dotenv
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI

load_dotenv()

Settings.llm = OpenAI(
    model=os.getenv("LLM_MODEL", "gpt-4o-mini"),
    api_key=os.getenv("OPENAI_API_KEY"),
    base_url=os.getenv("LLM_BASE_URL"),  # None falls back to OpenAI
    temperature=0.0,  # deterministic for legal work
)

Setting temperature=0.0 matters. Legal answers should not improvise.

Step 2: Load and chunk the document

LlamaIndex’s SimpleDirectoryReader handles PDFs via pypdf. For legal text, default chunking loses clause boundaries. Use a sentence splitter with a tight window.

from llama_index.core import SimpleDirectoryReader, Document
from llama_index.core.node_parser import SentenceSplitter

raw_docs = SimpleDirectoryReader("./data").load_data()
print(f"Loaded {len(raw_docs)} pages")

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = splitter.get_nodes_from_documents(raw_docs)
print(f"Created {len(nodes)} nodes")

Expected output:

Loaded 14 pages
Created 83 nodes

If you see far fewer nodes, your PDF is likely scanned images. Run OCR first (e.g., ocrmypdf) before ingestion.

Step 3: Build the vector index

We use the default in-memory vector store. For a single contract this is fine; for a corpus, swap in a persistent store like Qdrant.

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex(nodes)
index.storage_context.persist("./storage")

Persisting lets you skip re-embedding on restart. Embeddings default to text-embedding-3-small in current LlamaIndex.

Step 4: Query with citations

A legal document qa bot llamaindex ships without value if it can’t show where the answer came from. CitationQueryEngine forces the model to cite node IDs.

from llama_index.core.query_engine import CitationQueryEngine

cite_engine = CitationQueryEngine.from_index(
    index,
    similarity_top_k=4,
    citation_chunk_size=512,
)

response = cite_engine.query(
    "What are the termination conditions for the tenant?"
)
print(response.response)
for i, node in enumerate(response.source_nodes):
    print(f"[{i}] page {node.metadata.get('page_label')}: {node.text[:120]}...")

Sample output:

The tenant may terminate the lease early upon 30 days written notice if the landlord fails to remedy a material breach within 15 days [1][3].

[0] page 4: "12. Termination. (a) Tenant may terminate upon written notice..."
[1] page 5: "Material breach by Landlord includes failure to maintain habitable conditions..."
[2] page 4: "Notice must be delivered via certified mail to the address in Section 1."

The citations map to the printed source nodes. In a UI, link them to the PDF page offset.

Step 5: Validate retrieval before trusting answers

LLM citation can still hallucinate if retrieval is weak. Inspect what the retriever returns for a known clause.

retriever = index.as_retriever(similarity_top_k=4)
nodes = retriever.retrieve("late rent penalty")
for n in nodes:
    print(round(n.score, 3), n.metadata.get("page_label"), n.text[:80])

Expected: highest scores on pages describing penalties, not the signature block. If scores are flat (<0.2 spread), tighten chunk_size or use a legal-specific embedding model.

Step 6: Add metadata filtering for multi-document sets

When the legal document qa bot llamaindex grows beyond one file, filter by document type or jurisdiction. Attach metadata at load time:

for d in raw_docs:
    d.metadata["jurisdiction"] = "CA"
    d.metadata["doc_type"] = "lease"

nodes = splitter.get_nodes_from_documents(raw_docs)
index = VectorStoreIndex(nodes)

Then query with a filter:

from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

filters = MetadataFilters(
    filters=[MetadataFilter(key="doc_type", value="lease")]
)
filtered_engine = cite_engine.with_filters(filters)

This prevents a query about “termination” from pulling a divorce decree in the same index.

Step 7: Production hardening

The prototype above breaks under real load in three ways:

1. Rate limits. OpenAI returns 429s during bulk ingestion. Wrap index builds with retry, or route through a gateway that fails over.
2. Cost leakage. Long contracts explode token usage with similarity_top_k=8. Keep top_k at 3–4 for narrow clauses.
3. No audit trail. Log every query with the node IDs returned. Legal users need reproducibility.

A minimal retry wrapper for ingestion:

import time
from llama_index.core import VectorStoreIndex

def build_index_with_retry(nodes, attempts=3):
    for i in range(attempts):
        try:
            return VectorStoreIndex(nodes)
        except Exception as e:
            if "429" in str(e) and i < attempts - 1:
                time.sleep(2 ** i)
                continue
            raise
    return None

For serving, expose the query engine behind a FastAPI route and return the citation list as structured JSON, not just text.

Final notes on the legal document qa bot llamaindex pattern

The architecture we built—sentence-split ingestion, citation-forced querying, metadata filters—transfers to healthcare and finance docs with only embedding and chunk-size tweaks. The legal document qa bot llamaindex codebase should treat the LLM as a constrained renderer over retrieved text, never as the source of truth. Keep the vector store warm, log the nodes, and pin the model version.