Chunking strategies for LangChain RAG pipelines

Poor chunking silently ruins LangChain RAG retrieval long before the embedding model or vector database gets blamed. The right langchain rag chunking strategies match split boundaries to your document structure and downstream query patterns, not just to a fixed character count.

Why chunking decides retrieval quality

Embeddings map a chunk of text to a vector; that vector represents the whole chunk’s meaning. If you split a sentence mid-thought, the embedding blends two unrelated ideas and retrieval recalls the wrong neighbors. If you pack 5,000 tokens into one chunk, the vector averages everything and loses specificity.

Retrieval-augmented generation lives or dies on precision at the chunk level. You can swap embedding models or upgrade your vector DB, but if chunks are garbage, answers are garbage. Chunk size also dictates how many tokens you pay to embed, store, and later retrieve.

Start with RecursiveCharacterTextSplitter

For 90% of prototypes, RecursiveCharacterTextSplitter is the right default. It tries to split on paragraphs, then lines, then sentences, then words, preserving structure where possible.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    length_function=len,
    separators=["\n\n", "\n", ". ", " "]
)

docs = splitter.create_documents([raw_text])

The separators list is ordered by priority. The splitter recurses: if a paragraph exceeds chunk_size, it falls to the next separator.

Pitfall: length_function=len counts characters, not tokens. If your embedding model uses a BPE tokenizer (e.g., OpenAI text-embedding-3-small), character counts lie. Use tiktoken or a LangChain token counter for accurate sizing:

import tiktoken

def tok_len(text: str) -> int:
    enc = tiktoken.encoding_for_model("text-embedding-3-small")
    return len(enc.encode(text))

splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64, length_function=tok_len)

Fixed-size chunking with overlap

When documents have no natural structure (logs, flat strings), fixed-size splitting is simplest:

from langchain.text_splitter import CharacterTextSplitter

splitter = CharacterTextSplitter(
    separator=" ",
    chunk_size=256,
    chunk_overlap=32,
    length_function=len
)

Tradeoff: you will cut words and sentences. Overlap mitigates context loss but inflates storage and embedding cost linearly. Keep overlap between 10–20% of chunk size; more than that rarely helps and can duplicate facts in context.

Semantic chunking with embeddings

Semantic chunking uses embedding similarity to detect topic shifts. langchain_experimental ships SemanticChunker:

from langchain_experimental.text_splitter import SemanticChunker
from langchain.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
chunker = SemanticChunker(embeddings, breakpoint_threshold_type="percentile", breakpoint_threshold_amount=95)

semantic_docs = chunker.create_documents([raw_text])

The splitter embeds each sentence, computes cosine distance to the next, and breaks where distance spikes. This aligns chunks to meaning, not whitespace.

When generating these embeddings, point your wrapper at an OpenAI-compatible endpoint—n4n.ai exposes one that covers 240+ models with automatic fallback if a provider is rate-limited, which keeps chunking pipelines stable during batch jobs.

Threshold types

breakpoint_threshold_type accepts percentile, standard_deviation, or interquartile. Percentile 95 works on uniform prose; standard deviation suits noisy transcripts. Tune on a sample before running the full corpus.

Tradeoffs: semantic chunking adds one embedding call per sentence. On a 100-page PDF that’s thousands of inferences before you even store a vector. It also needs threshold tuning; a fixed percentile may over-split terse documents.

Structure-aware splitting for Markdown and HTML

Technical docs often ship as Markdown or HTML. Use header splitters to keep section context:

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers = [("#", "h1"), ("##", "h2"), ("###", "h3")]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
md_docs = splitter.split_text(md_text)

Each output doc carries metadata with its header path. That metadata is gold for filtering later (“only search under ## API Reference”).

HTML variant works the same with HTMLHeaderTextSplitter. Pitfall: auto-generated docs often have nested tables or code blocks that header splitters ignore. You still need a secondary character splitter on the leaf nodes.

Parent-document and small-to-big retrieval

Embed small, retrieve big. Split documents into small chunks for embedding, but keep a pointer to the parent larger chunk for generation context.

from langchain.retrievers import ParentDocumentRetriever
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.storage import InMemoryStore

child_splitter = RecursiveCharacterTextSplitter(chunk_size=128)
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=1024)

vectorstore = Chroma(embedding_function=OpenAIEmbeddings())
store = InMemoryStore()

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_document_splitter=child_splitter,
    parent_document_splitter=parent_splitter,
)
retriever.add_documents(docs)

The retriever embeds children, but returns the parent on match. This avoids truncated context in the prompt while keeping embedding precision.

Docstore backends

InMemoryStore is fine for scripts. For production, use RedisStore or MongoDBStore so the parent docs survive process restarts. The vector DB holds only child embeddings; the docstore holds the raw text.

Tuning chunk size and overlap

Empirical starting points:

• Chunk size: 512–1024 tokens for prose; 128–256 for dense API docs.
• Overlap: 10–20% of chunk size.
• Separator priority: paragraph > line > sentence > word.

Measure with a labeled eval set: take 50 real queries, check if the gold passage is fully contained in the top-k retrieved chunks. If the gold passage is split across two chunks, increase overlap or switch splitter.

Query-aware and late chunking

Late chunking (embed the whole doc, then slice attention) is emerging but not native in LangChain yet. For query-aware splitting, do a first-pass retrieval on coarse chunks, then re-split the matched region with finer granularity before the final prompt. This adds latency but boosts precision for long documents.

Common pitfalls

• Ignoring tokenizer mismatch: counting characters when the model counts tokens yields oversized chunks that get truncated silently.
• Dropping metadata: splitters preserve metadata, but custom post-processing often drops it. Keep source, page, and header in the stored vector.
• Overlapping too much: 50% overlap doubles cost and can cause the same fact to appear twice in context, confusing the LLM.
• One-size-fits-all: a legal contract and a Python repo need different separators. Configure per document type.
• Skipping eval: intuition about “good” chunks fails. Always run retrieval recall on real questions.

Actionable ordered path

1. Parse raw bytes into text per type (PDF, MD, HTML).
2. Apply MarkdownHeaderTextSplitter or HTMLHeaderTextSplitter to preserve structure.
3. Run RecursiveCharacterTextSplitter on leaf blocks with token-accurate length_function.
4. For ambiguous docs, add a SemanticChunker pass on large leaves.
5. Store vectors from child chunks; keep parent docs in a docstore.
6. Evaluate retrieval with real queries; tune chunk_size, overlap, and separators.
7. Add metadata filters to the vector DB to narrow by header or source.

Good langchain rag chunking strategies are iterative. Ship the recursive splitter first, measure, then reach for semantic or structural methods only where the eval shows misses.