Building a hipaa document qa langchain pipeline forces you to treat every retrieved chunk and generated token as potential protected health information. This tutorial walks through a concrete implementation that loads clinical PDFs, embeds them locally, and answers questions through a compliant model endpoint with redaction guardrails.
Prerequisites
- Python 3.10 or newer
- Packages:
langchain,langchain-community,langchain-openai,chromadb,pypdf,python-dotenv - An LLM endpoint covered by a Business Associate Agreement (BAA). If you use a third-party gateway, confirm it forwards to BAA-backed providers only. For example, an OpenAI-compatible gateway like n4n.ai can route across 240+ models with automatic fallback when a provider is degraded, but you remain responsible for the BAA.
- A test PDF containing synthetic PHI. Never point this code at real patient records without a full compliance review.
Set up a virtual environment and install dependencies:
pip install langchain langchain-community langchain-openai chromadb pypdf python-dotenv1. Load and split the document
Use PyPDFLoader to pull text, then split into chunks small enough for embedding and retrieval. Keep chunk size conservative to limit PHI exposure per retrieval.
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
loader = PyPDFLoader("synthetic_clinical_note.pdf")
pages = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
separators=["\n\n", "\n", ". ", " "]
)
docs = splitter.split_documents(pages)
print(f"Loaded {len(docs)} chunks")Expected output:
Loaded 42 chunks2. Embed and store locally
Embeddings run against your chosen model. Store vectors in Chroma on local disk so PHI never leaves your infrastructure unencrypted. The hipaa document qa langchain pattern depends on this swapability to avoid vendor lock-in.
import os
from dotenv import load_dotenv
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
load_dotenv()
os.environ["OPENAI_API_BASE"] = os.getenv("LLM_API_BASE")
os.environ["OPENAI_API_KEY"] = os.getenv("LLM_API_KEY")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
documents=docs,
embedding=embeddings,
persist_directory="./chroma_db"
)
vectorstore.persist()If your LLM_API_BASE points to a compliant gateway, the same code works without changes.
3. Build the retriever and chat model
Create a retriever that returns the top 3 chunks. Use a chat model with temperature 0 for deterministic answers.
from langchain_openai import ChatOpenAI
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)4. Add HIPAA guardrails
Never trust the model to spontaneously omit PHI. Apply regex redaction on both the retrieved context (for logging) and the final answer. This is a baseline; pair it with a human review step in production.
import re
PHI_PATTERNS = [
(r"\b\d{3}-\d{2}-\d{4}\b", "[SSN]"),
(r"\b\d{3}-\d{3}-\d{4}\b", "[PHONE]"),
(r"\b\d{1,2}/\d{1,2}/\d{4}\b", "[DATE]"),
]
def redact(text: str) -> str:
for pattern, replacement in PHI_PATTERNS:
text = re.sub(pattern, replacement, text)
return textWrap the LLM call so the answer is scrubbed before it reaches the caller:
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=retriever,
return_source_documents=True
)
def ask(query: str) -> str:
raw = qa.invoke({"query": query})
answer = redact(raw["result"])
return answer5. Run the Q&A loop
Query for a specific patient detail. The system retrieves chunks, generates an answer, and redacts identifiers.
question = "What medication was prescribed for the patient with SSN 123-45-6789?"
answer = ask(question)
print(answer)Expected output (synthetic data):
The prescribed medication was Lisinopril 10mg daily. The patient's identifier [SSN] has been redacted per policy.Source documents are available in raw["source_documents"]; redact them before any logging.
6. Conversation memory with isolation
For multi-turn Q&A, use ConversationalRetrievalChain but disable persistent memory on disk. Keep the chat history in memory only and clear it after the session.
from langchain.memory import ConversationBufferMemory
from langchain.chains import ConversationalRetrievalChain
memory = ConversationBufferMemory(
memory_key="chat_history",
return_messages=True,
output_key="answer"
)
crc = ConversationalRetrievalChain.from_llm(
llm=llm,
retriever=retriever,
memory=memory
)
def chat(query: str) -> str:
res = crc.invoke({"question": query})
return redact(res["answer"])The hipaa document qa langchain stack must isolate session state. Never write chat_history to a file without encryption and a retention policy.
7. Deployment notes
The pipeline is only compliant if the surrounding infrastructure is. Encrypt the Chroma directory at rest with dm-crypt or cloud KMS. Set OPENAI_API_KEY from a secrets manager, not a .env file in the repo. Disable LangChain verbose logging in production—it prints retrieved documents.
If you use a gateway, confirm it meters per-token usage and forwards provider cache-control hints so you can cache embeddings and common retrievals. That reduces cost and limits repeated PHI transmission.
Finally, add an access control layer: issue short-lived tokens to clinicians, and audit every query. The code above is a starting point, not a turnkey compliant product.
8. Unit test the redaction
Guardrails fail silently if regexes drift. Add a small test to lock behavior:
def test_redact():
sample = "Patient 123-45-6789 called 555-123-4567 on 01/02/2023."
cleaned = redact(sample)
assert "[SSN]" in cleaned
assert "[PHONE]" in cleaned
assert "[DATE]" in cleaned
assert "123-45-6789" not in cleanedRun pytest test_redact.py. Expected output:
1 passed in 0.02sWhat you built
You now have a runnable pipeline that ingests PHI documents, retrieves context, and answers with redaction. The hipaa document qa langchain approach keeps embedding and vector storage local, routes generation to a BAA-covered model, and scrubs output. Extend it with entity recognition for finer redaction and a signed audit log before production use.