n4nAI
Open console →
Framework tutorialsGuideLangChain Document Loaders & Chunking

LangChain document loaders: PDF, HTML, and Markdown

Practical guide to langchain document loaders pdf html markdown: install, load each format with code, dodge pitfalls, and chunk for retrieval.

n4n Team3 min read700 words

Audio narration

Coming soon — every post will get a voice note here.

Most RAG prototypes die at the data ingestion step. The langchain document loaders pdf html markdown trio handles the three formats you’ll hit in every corporate knowledge base, but each has sharp edges that can silently corrupt your context window. This guide walks an ordered path from install to chunked documents ready for embedding.

Installation and dependencies

LangChain moved its loaders into langchain_community as of v0.1. You need that package plus format-specific native libraries.

pip install langchain-community pypdf beautifulsoup4

If you prefer the unstructured backends for smarter HTML/PDF partitioning, add unstructured[pdf] and unstructured[html]. That pulls heavy system deps (libreoffice, poppler). For most teams, the lightweight loaders below are enough.

Version pinning

Pin versions in production. langchain-community moves fast; a minor bump changed the BSHTMLLoader signature in 0.2. Use a lockfile and test loader imports in CI.

Loading Markdown

Markdown is the easiest format, but “easy” hides two failure modes: encoding and unrendered syntax. The langchain document loaders pdf html markdown set treats .md as plain text unless you pick a structured backend.

from langchain_community.document_loaders import TextLoader

loader = TextLoader("spec.md", encoding="utf-8")
docs = loader.load()
print(f"Loaded {len(docs)} doc(s), {len(docs[0].page_content)} chars")

TextLoader returns the raw file. If your Markdown uses frontmatter (Jekyll, Hugo), the YAML block stays in page_content. Strip it pre-load or use a custom loader.

For structured extraction, UnstructuredMarkdownLoader partitions by headings:

from langchain_community.document_loaders import UnstructuredMarkdownLoader

loader = UnstructuredMarkdownLoader("spec.md")
docs = loader.load()
# each heading section may become its own Document if unstructured is installed

Tradeoff: UnstructuredMarkdownLoader needs the unstructured package and is slower. For a simple RAG over a docs site, TextLoader plus a regex split on ^## is often cheaper.

Pitfall: Windows cp1252 files. Always pass encoding="utf-8"; the default is platform-dependent and will raise UnicodeDecodeError on em dashes.

Loading HTML

HTML carries navigation, ads, and script tags. BSHTMLLoader uses BeautifulSoup to extract <body> text.

from langchain_community.document_loaders import BSHTMLLoader

loader = BSHTMLLoader("page.html", open_encoding="utf-8")
docs = loader.load()
print(docs[0].metadata)  # includes source path

The loader strips <script>, <style>, and comments. It does not remove boilerplate like nav bars. If your pages are template-heavy, add a post-filter:

def remove_nav(text: str) -> str:
    return "\n".join(l for l in text.splitlines() if "menu" not in l.lower())
docs[0].page_content = remove_nav(docs[0].page_content)

For higher fidelity, UnstructuredHTMLLoader partitions by DOM structure and emits elements with metadata types (Title, NarrativeText). That helps chunking but costs the unstructured dependency.

Pitfall: relative links and embedded base64 images bloat the text. Pre-process with an HTML sanitizer if needed.

Loading PDF

PDF is where most teams get burned. PyPDFLoader is the baseline:

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("report.pdf")
pages = loader.load()  # list of Document, one per page
print(f"Pages: {len(pages)}, first page chars: {len(pages[0].page_content)}")

Each Document carries page in metadata. For single-file embedding, that’s fine. For semantic chunking, page breaks cut sentences.

Tradeoffs:

  • PyPDFLoader reads text layers. Scanned PDFs return empty strings; you need OCR or a vision model.
  • Multi-column layouts are read left-to-right across columns, scrambling reading order.
  • Tables become garbled text.

If layout matters, use UnstructuredPDFLoader with mode="elements":

from langchain_community.document_loaders import UnstructuredPDFLoader

loader = UnstructuredPDFLoader("report.pdf", mode="elements")
docs = loader.load()

This returns separate Documents for titles, tables, and text, but requires unstructured and poppler.

Pitfall: PDFs with rotated pages produce weird whitespace. Normalize with pdf2image + OCR if accuracy is critical.

Common pitfalls across langchain document loaders pdf html markdown

All three share failure modes:

  1. Metadata starvation – Loaders set only source. Add created_at, author, doc_type before embedding; your retriever will thank you.
  2. Silent empty docs – Check if not doc.page_content.strip(): and log. A 0-byte file passes load() silently.
  3. Encoding mismatches – UTF-8 is not universal. Pass explicit encoding.
  4. Path traversal – If loading user-supplied paths, sanitize. Loaders will read anything the OS permits.

Enrich metadata cheaply:

import os
for d in docs:
    d.metadata["size"] = os.path.getsize(d.metadata["source"])

Chunking the loaded documents

Loading is half the battle. You must split for embedding models’ token limits.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
    separators=["\n## ", "\n### ", "\n", " ", ""]
)

all_chunks = []
for d in docs:
    all_chunks.extend(splitter.split_documents([d]))

For Markdown, the \n## separator keeps sections intact. For HTML, consider splitting by extracted heading elements from UnstructuredHTMLLoader. For PDF, page boundaries plus recursive split works.

Tradeoff: larger chunk_size improves context but hurts retrieval precision. Start at 512–1024 tokens and tune with eval.

Streaming large corpora

If you load thousands of files, load() buffers everything in memory. Use lazy_load():

for doc in loader.lazy_load():
    chunks = splitter.split_documents([doc])
    embed_and_store(chunks)

This yields Documents one at a time, letting you chunk and embed incrementally. Works for all three loader types.

Serving chunks to a model

Once chunked and embedded, you retrieve and send context to an LLM. If you serve the retrieved chunks through n4n.ai, the gateway’s per-token metering and automatic fallback keep provider outages from breaking your pipeline while honoring client routing directives. The OpenAI-compatible endpoint means you swap openai for n4n in your client with one line.

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": retrieved_context}]
)

That’s the full path: load, clean, chunk, retrieve, generate.

Quick reference

  • Markdown: TextLoader (light) or UnstructuredMarkdownLoader (structured)
  • HTML: BSHTMLLoader (fast) or UnstructuredHTMLLoader (partitioned)
  • PDF: PyPDFLoader (text layer) or UnstructuredPDFLoader (layout aware)

Pick the lightest loader that meets your accuracy bar; you can always upgrade later. The langchain document loaders pdf html markdown APIs are stable enough to build on, but verify output on real files before trusting them in production.

Tagslangchaindocument-loaderspdfhtml

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Framework tutorialsLoading PDFs into LangChain with PyPDFLoaderLearn to load PDFs into LangChain using PyPDFLoader with step-by-step code examples, metadata extraction, text splitting, and error handling for production use.Framework tutorialsLangChain CSV and JSON loaders for structured dataPractical guide to loading CSV and JSON files with LangChain document loaders, including nested structures, custom parsing, and chunking strategies for RAG pipelines.Framework tutorialsLangChain document loaders for Google Drive and S3A step-by-step guide to loading documents from Google Drive and Amazon S3 using LangChain document loaders, with runnable code and verification steps.Framework tutorialsWeb scraping documents into LangChain with WebBaseLoaderA practical guide to scraping web documents into LangChain using WebBaseLoader, covering authentication, JavaScript rendering, custom parsing, and pipeline integration.Framework tutorialsLangChain document loaders for Notion and ConfluenceA practical guide to loading Notion and Confluence content into LangChain with working code, chunking strategies, and verification steps.Framework tutorialsSemantic chunking in LangChain with embeddingsLearn to implement semantic chunking in LangChain using embeddings for better RAG retrieval — step-by-step with runnable code and evaluation techniques.

All langchain document loaders & chunking posts →