Chunking strategies for PDFs and long documents

Chunking PDFs for RAG is the step where most retrieval pipelines quietly fail. You can have the best embedding model and a perfectly tuned reranker, but if your chunks split mid-sentence, orphan table headers, or drown signal in noise, retrieval quality collapses. This guide walks through an ordered path from naive splitting to production-grade strategies, with code you can adapt and the tradeoffs you’ll actually face.

Start with the constraints, not the library

Before picking a splitter, define what “good” looks like for your corpus and queries. Three constraints drive every decision:

Token budget per chunk. Your embedding model has a max context (usually 512–8192 tokens). Your LLM context window limits how many chunks you can stuff into a prompt. Target 256–1024 tokens per chunk for most dense retrievers; go smaller (128–256) if you’re using late-interaction models like ColBERT or need fine-grained citation.

Semantic coherence. A chunk should answer a standalone question. If a query matches a chunk but the answer requires the previous paragraph, you’ve failed. This means respecting document structure — headings, lists, tables, code blocks — not just character counts.

Overlap strategy. Overlap prevents boundary losses but duplicates tokens and inflates index size. Fixed overlap (e.g., 100 tokens) is simple; semantic overlap (carry the last complete sentence or paragraph) is better but harder to implement correctly.

The naive baseline: recursive character splitting

Most teams start here. LangChain’s RecursiveCharacterTextSplitter (or its equivalents in LlamaIndex, Haystack) splits by a prioritized list of separators: ["\n\n", "\n", ". ", " ", ""]. It’s fast, dependency-light, and works acceptably for plain text.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_text(raw_text)

Pitfall: Character count ≠ token count. A 1000-character chunk can be 150 tokens or 400 depending on language, code density, and whitespace. Always measure with your actual tokenizer.

Pitfall: PDFs extracted via PyPDF2 or pdfplumber produce garbage whitespace — headers/footers repeated on every page, column breaks mid-sentence, ligature artifacts. Recursive splitting on raw extraction output propagates all of it.

Extract structure first, then chunk

PDFs are not text files. They’re layout engines. Treat extraction and chunking as separate stages.

Use a layout-aware extractor

marker (by VikParuchuri), pymupdf4llm, or unstructured.io preserve heading hierarchy, table structure, and reading order. They output markdown or structured JSON, not flat text.

import fitz  # pymupdf
from pymupdf4llm import to_markdown

md_text = to_markdown("doc.pdf")  # preserves # headings, tables as markdown

Chunk by heading hierarchy

Once you have markdown, split on heading boundaries first. This keeps sections intact and gives you metadata for filtering (e.g., “only retrieve from Section 3.2”).

import re
from dataclasses import dataclass
from typing import List

@dataclass
class Chunk:
    text: str
    heading_path: List[str]  # ["Chapter 2", "2.3 Methods"]
    token_count: int
    page_start: int
    page_end: int

def chunk_by_headings(md: str, max_tokens: int = 512, overlap_tokens: int = 50) -> List[Chunk]:
    # Split on ATX headings (#, ##, ###)
    heading_pattern = re.compile(r'^(#{1,6})\s+(.+)$', re.MULTILINE)
    matches = list(heading_pattern.finditer(md))
    
    chunks = []
    for i, match in enumerate(matches):
        level = len(match.group(1))
        title = match.group(2).strip()
        start = match.end()
        end = matches[i + 1].start() if i + 1 < len(matches) else len(md)
        section_text = md[start:end].strip()
        
        # Build heading path from ancestor headings
        path = [m.group(2).strip() for m in matches[:i+1] if len(m.group(1)) <= level]
        
        # If section too large, sub-chunk recursively
        if estimate_tokens(section_text) > max_tokens:
            sub_chunks = recursive_subchunk(section_text, max_tokens, overlap_tokens)
            for sc in sub_chunks:
                chunks.append(Chunk(sc, path, estimate_tokens(sc), 0, 0))
        else:
            chunks.append(Chunk(section_text, path, estimate_tokens(section_text), 0, 0))
    
    return chunks

Tradeoff: Heading-based chunking fails on documents without clear structure (scanned reports, forms, messy invoices). Have a fallback.

Handle tables and code blocks as first-class citizens

Tables and code are high-signal, high-fragility. Splitting a table across chunks destroys its utility. Splitting a function mid-block loses context.

Detect and preserve blocks

Markdown extractors emit fenced code blocks and pipe tables. Parse them out, keep them whole, and attach metadata.

BLOCK_PATTERN = re.compile(
    r'(```[\s\S]*?```|^\|.*\|(?:\n\|.*\|)*$)',
    re.MULTILINE
)

def extract_blocks(text: str) -> List[tuple[str, str, int, int]]:
    """Returns [(block_type, block_text, start, end), ...]"""
    blocks = []
    for match in BLOCK_PATTERN.finditer(text):
        block_text = match.group(0)
        btype = "code" if block_text.startswith("```") else "table"
        blocks.append((btype, block_text, match.start(), match.end()))
    return blocks

def chunk_preserving_blocks(text: str, max_tokens: int) -> List[str]:
    blocks = extract_blocks(text)
    if not blocks:
        return recursive_subchunk(text, max_tokens, 50)
    
    chunks = []
    last_end = 0
    buffer = ""
    
    for btype, btext, start, end in blocks:
        # Flush text before this block
        if start > last_end:
            prefix = text[last_end:start]
            buffer += prefix
            if estimate_tokens(buffer) >= max_tokens:
                chunks.append(buffer.strip())
                buffer = ""
        
        # Handle the block itself
        block_tokens = estimate_tokens(btext)
        if block_tokens > max_tokens:
            # Block too large — truncate with warning, or split intelligently
            if btype == "code":
                chunks.append(truncate_code_block(btext, max_tokens))
            else:
                chunks.append(truncate_table(btext, max_tokens))
        else:
            if estimate_tokens(buffer + btext) > max_tokens:
                chunks.append(buffer.strip())
                buffer = btext
            else:
                buffer += btext
        
        last_end = end
    
    # Flush remainder
    if buffer.strip():
        chunks.append(buffer.strip())
    
    return chunks

Pitfall: Large tables (100+ rows) exceed any reasonable chunk size. Options: (a) summarize the table with an LLM and store both summary + full table in metadata, (b) chunk by row groups with header repetition, (c) use a table-specific retriever (like TAPAS) for structured queries. Option (a) is the most pragmatic for general RAG.

Semantic chunking: when structure isn’t enough

For unstructured prose — legal contracts, research papers, long-form articles — heading-based splitting still produces chunks that mix unrelated concepts. Semantic chunking uses embedding similarity to find natural boundaries.

The algorithm

Split into sentences (or small fixed windows).
Embed each unit.
Compute cosine similarity between adjacent units.
Cut where similarity drops below a threshold (or at local minima).

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def semantic_chunk(text: str, threshold: float = 0.75, max_chunk_tokens: int = 512) -> List[str]:
    sentences = split_sentences(text)  # use a real sentence tokenizer
    if len(sentences) &lt;= 1:
        return [text]
    
    embeddings = model.encode(sentences, normalize_embeddings=True)
    similarities = np.dot(embeddings[:-1], embeddings[1:].T).diagonal()
    
    boundaries = [0]
    for i, sim in enumerate(similarities):
        if sim &lt; threshold:
            boundaries.append(i + 1)
    boundaries.append(len(sentences))
    
    chunks = []
    for i in range(len(boundaries) - 1):
        chunk_sents = sentences[boundaries[i]:boundaries[i+1]]
        chunk_text = " ".join(chunk_sents)
        # Enforce hard token limit
        if estimate_tokens(chunk_text) > max_chunk_tokens:
            chunks.extend(recursive_subchunk(chunk_text, max_chunk_tokens, 50))
        else:
            chunks.append(chunk_text)
    
    return chunks

Tradeoff: Semantic chunking is slower (embedding pass) and non-deterministic (model-dependent boundaries). It also tends to produce variable chunk sizes, which complicates batching. Use it for prose-heavy corpora; skip for structured docs where headings already capture semantics.

Pitfall: Threshold tuning is corpus-specific. 0.75 works for technical writing; narrative text may need 0.65. Evaluate on a held-out set with retrieval metrics, not vibes.

Parent-child retrieval: the best of both worlds

Small chunks retrieve precisely but lose context. Large chunks preserve context but dilute signal. Parent-child (or “small-to-big”) indexing solves this: embed small chunks for retrieval, but return their parent large chunk for generation.

from dataclasses import dataclass
from typing import List

@dataclass
class ParentChunk:
    id: str
    text: str
    child_ids: List[str]
    metadata: dict

@dataclass
class ChildChunk:
    id: str
    text: str
    parent_id: str
    metadata: dict

def build_parent_child(chunks: List[Chunk], child_size: int = 256, parent_size: int = 1024) -> tuple[List[ParentChunk], List[ChildChunk]]:
    parents = []
    children = []
    
    for i, chunk in enumerate(chunks):
        parent_id = f"parent_&#123;i&#125;"
        # Parent is the full section (or merged adjacent sections up to parent_size)
        parent_text = chunk.text
        parents.append(ParentChunk(parent_id, parent_text, [], chunk.metadata))
        
        # Children are fixed-size splits of the parent
        child_texts = recursive_subchunk(parent_text, child_size, 50)
        for j, ctext in enumerate(child_texts):
            child_id = f"&#123;parent_id&#125;_child_&#123;j&#125;"
            children.append(ChildChunk(child_id, ctext, parent_id, chunk.metadata))
            parents[-1].child_ids.append(child_id)
    
    return parents, children

Retrieval flow:

Embed query → search child index (vector DB)
Collect unique parent IDs from top-k children
Fetch full parent texts → rerank → pass to LLM

Tradeoff: Double the index size (roughly). Requires a vector DB that supports metadata filtering or a separate key-value store for parent lookup. Most managed services (Pinecone, Weaviate, Qdrant) handle this natively.

Metadata is not optional

Every chunk needs: source_doc_id, page_range, heading_path, chunk_index, token_count, extraction_method. At query time, you’ll filter by doc ID, boost recent pages, or cite page numbers. Without metadata, you’re guessing.

# Minimal metadata schema
CHUNK_METADATA_SCHEMA = &#123;
    "doc_id": "str",
    "doc_title": "str",
    "page_start": "int",
    "page_end": "int",
    "heading_path": "list[str]",
    "chunk_index": "int",
    "token_count": "int",
    "chunk_strategy": "str",  # "heading", "semantic", "recursive"
    "extractor": "str",       # "pymupdf4llm", "marker", "unstructured"
    "created_at": "datetime"
&#125;

Pitfall: Storing full heading paths in every chunk duplicates strings. Normalize: store heading IDs in the chunk, keep a separate heading lookup table. Matters at millions of chunks.

Evaluate your chunking, don’t guess

Chunking quality is measurable. Build a small eval set: 50–100 questions with ground-truth answer spans (page + paragraph). Measure:

Recall@k: Does the correct chunk appear in top-k?
MRR: How high is the first relevant chunk?
Context sufficiency: Given the retrieved chunks, can an LLM actually answer? (Use a judge model or human eval.)

def evaluate_chunking(questions: List[dict], retriever, k: int = 10) -> dict:
    recalls = []
    mrrs = []
    
    for q in questions:
        results = retriever.search(q["question"], k=k)
        retrieved_ids = [r.metadata["chunk_id"] for r in results]
        relevant = set(q["relevant_chunk_ids"])
        
        # Recall@k
        recalls.append(len(relevant & set(retrieved_ids)) / len(relevant))
        
        # MRR
        for rank, cid in enumerate(retrieved_ids, 1):
            if cid in relevant:
                mrrs.append(1.0 / rank)
                break
        else:
            mrrs.append(0.0)
    
    return &#123;
        "recall_at_k": np.mean(recalls),
        "mrr": np.mean(mrrs),
        "per_question": list(zip(recalls, mrrs))
    &#125;

Run this after every chunking strategy change. A 5% recall drop from a “clever” semantic splitter isn’t worth it.

Common failure modes checklist

Symptom	Likely Cause	Fix
Retrieval returns table header without rows	Recursive split on markdown table	Extract tables as atomic blocks
Answer needs previous section context	Heading chunks too small, no overlap	Increase parent size, use parent-child
High recall but LLM hallucinates	Chunks contain contradictory fragments	Stricter semantic boundaries, dedupe
Index size explodes	Fixed 100-token overlap on 1M docs	Semantic overlap, or parent-child with small children
Slow ingestion	Semantic chunking embeds every sentence	Cache embeddings, batch, or skip for structured docs

Putting it together: a production pipeline

def ingest_pdf(path: str, doc_id: str) -> List[Chunk]:
    # 1. Extract with layout awareness
    md = to_markdown(path)  # pymupdf4llm or marker
    
    # 2. Try heading-based chunking
    chunks = chunk_by_headings(md, max_tokens=512)
    
    # 3. Fallback for heading-less sections
    final_chunks = []
    for chunk in chunks:
        if chunk.token_count > 512:
            # Sub-chunk with block preservation
            sub = chunk_preserving_blocks(chunk.text, 512)
            for i, s in enumerate(sub):
                final_chunks.append(Chunk(
                    text=s,
                    heading_path=chunk.heading_path,
                    token_count=estimate_tokens(s),
                    page_start=chunk.page_start,
                    page_end=chunk.page_end
                ))
        else:
            final_chunks.append(chunk)
    
    # 4. Attach metadata
    for i, chunk in enumerate(final_chunks):
        chunk.metadata = &#123;
            "doc_id": doc_id,
            "chunk_index": i,
            "heading_path": chunk.heading_path,
            "token_count": chunk.token_count,
            "page_start": chunk.page_start,
            "page_end": chunk.page_end,
            "chunk_strategy": "heading+block",
            "extractor": "pymupdf4llm"
        &#125;
    
    return final_chunks

This pipeline handles 90% of real-world PDFs: structured reports, papers, manuals. The remaining 10% (scans, forms, multi-column chaos) need OCR + layout analysis (Marker, Unstructured, or cloud APIs) — but the chunking logic stays the same.

One final constraint: the gateway

If your retrieval pipeline sits behind an inference gateway that routes across providers, chunking affects routing efficiency. Smaller chunks mean more embedding calls (cheap) but more vector DB reads (latency). Larger chunks mean fewer reads but more tokens into the LLM (cost). The gateway can’t fix a bad chunking strategy, but it can expose per-request token accounting so you see the tradeoff in production. n4n.ai surfaces per-token usage per model so you can correlate chunk size with actual cost — not estimates.

Next steps: Build the eval set first. Then implement heading+block chunking as your baseline. Only add semantic splitting or parent-child when metrics demand it. Most teams over-engineer chunking and under-invest in evaluation.