Chunking is the single most impactful preprocessing decision in a Haystack RAG pipeline. Get it wrong and you either drown the model in noise or starve it of context. This guide walks through the major strategies, shows working Haystack code for each, and gives you a framework for choosing and validating the right approach for your corpus.
Why chunking strategy matters
Retrieval quality is bounded by chunk quality. If a chunk splits a logical unit — a code function, a legal clause, a product spec — the embedding captures a fractured idea. If chunks are too large, you waste context window on irrelevant text and dilute the signal. If they’re too small, you lose the relationships that make the answer coherent.
Haystack’s DocumentSplitter component handles the mechanics, but the strategy is yours. The right choice depends on document type, query patterns, and your tolerance for latency versus recall. We’ll cover four approaches in order of increasing sophistication.
Fixed-size chunking with overlap
The baseline. Split by character or token count with a sliding window. Simple, deterministic, and fast. Works surprisingly well for homogeneous prose — blog posts, news articles, documentation with consistent structure.
from haystack.components.preprocessors import DocumentSplitter
splitter = DocumentSplitter(
split_by="word",
split_length=250,
split_overlap=50,
split_threshold=10 # don't create tiny trailing chunks
)
# In a pipeline
from haystack import Pipeline
indexing = Pipeline()
indexing.add_component("converter", PyPDFToDocument())
indexing.add_component("splitter", splitter)
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
indexing.add_component("writer", DocumentWriter(document_store))
indexing.connect("converter.documents", "splitter.documents")
indexing.connect("splitter.documents", "embedder.documents")
indexing.connect("embedder.documents", "writer.documents")Tradeoffs: Overlap mitigates boundary cuts but duplicates content — increasing index size and embedding cost by roughly overlap / (length - overlap). For 250/50 that’s 25% overhead. No semantic awareness means you’ll split mid-sentence, mid-table, mid-code-block. Acceptable for prototyping; rarely optimal for production.
Pitfall: split_by="word" uses whitespace tokenization, not model tokenization. A 250-word chunk can exceed 512 tokens for technical text. Use split_by="token" with a tokenizer if you’re pushing context limits:
from haystack.components.preprocessors import DocumentSplitter
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-small-en-v1.5")
splitter = DocumentSplitter(
split_by="token",
split_length=256,
split_overlap=32,
tokenizer=tokenizer
)Semantic chunking
Instead of fixed windows, split where meaning shifts. Haystack’s DocumentSplitter supports split_by="sentence" combined with embedding-based boundary detection via SemanticDocumentSplitter (available in haystack-experimental or as a custom component).
The algorithm: embed sentences, compute cosine similarity between adjacent sentences, split where similarity drops below a threshold.
# Custom semantic splitter component
from haystack import component, Document
from typing import List
import numpy as np
from sentence_transformers import SentenceTransformer
@component
class SemanticDocumentSplitter:
def __init__(
self,
model_name: str = "BAAI/bge-small-en-v1.5",
threshold: float = 0.65,
min_chunk_sentences: int = 3,
max_chunk_sentences: int = 20
):
self.model = SentenceTransformer(model_name)
self.threshold = threshold
self.min_chunk_sentences = min_chunk_sentences
self.max_chunk_sentences = max_chunk_sentences
@component.output_types(documents=List[Document])
def run(self, documents: List[Document]):
output_docs = []
for doc in documents:
sentences = self._split_sentences(doc.content)
if len(sentences) <= self.min_chunk_sentences:
output_docs.append(doc)
continue
embeddings = self.model.encode(sentences)
chunks = self._group_by_similarity(sentences, embeddings)
for i, chunk_sentences in enumerate(chunks):
chunk_content = " ".join(chunk_sentences)
meta = {**doc.meta, "chunk_id": i, "chunk_strategy": "semantic"}
output_docs.append(Document(content=chunk_content, meta=meta))
return {"documents": output_docs}
def _split_sentences(self, text: str) -> List[str]:
# Use a proper sentence tokenizer in production
import re
return [s.strip() for s in re.split(r'(?<=[.!?])\s+', text) if s.strip()]
def _group_by_similarity(self, sentences: List[str], embeddings: np.ndarray) -> List[List[str]]:
chunks = []
current_chunk = [sentences[0]]
for i in range(1, len(sentences)):
sim = np.dot(embeddings[i-1], embeddings[i]) / (
np.linalg.norm(embeddings[i-1]) * np.linalg.norm(embeddings[i])
)
if sim < self.threshold and len(current_chunk) >= self.min_chunk_sentences:
chunks.append(current_chunk)
current_chunk = [sentences[i]]
elif len(current_chunk) >= self.max_chunk_sentences:
chunks.append(current_chunk)
current_chunk = [sentences[i]]
else:
current_chunk.append(sentences[i])
if current_chunk:
chunks.append(current_chunk)
return chunksTradeoffs: Adapts to content. Keeps related sentences together, separates topic shifts. But: requires an embedding model at index time (extra latency), threshold is corpus-sensitive, and sentence splitting is brittle on messy text (PDFs, scanned docs). The min_chunk_sentences floor prevents over-fragmentation on low-similarity transitions like bullet lists.
When to use: Heterogeneous documents where fixed windows clearly break logical units — research papers, mixed-format knowledge bases, legal contracts with distinct clauses.
Document-structure-aware chunking
The highest signal comes from respecting the author’s structure: headings, sections, tables, code blocks. Haystack’s MarkdownDocumentSplitter (via haystack-experimental) or a custom HTML/PDF parser can preserve hierarchy.
from haystack.components.preprocessors import MarkdownDocumentSplitter
# Splits on markdown headers, preserves header path in metadata
splitter = MarkdownDocumentSplitter(
split_by="header",
split_level=2, # H2 boundaries
respect_code_blocks=True,
respect_tables=True
)
# Resulting chunks carry header hierarchy
# doc.meta = {"headers": {"h1": "API Reference", "h2": "Authentication"}, ...}For PDFs, combine a layout-aware extractor (like PyMuPDF or pdfplumber) with structure detection:
import pdfplumber
from haystack import Document
def extract_structured_chunks(pdf_path: str) -> List[Document]:
docs = []
with pdfplumber.open(pdf_path) as pdf:
for page_num, page in enumerate(pdf.pages):
# Extract tables separately
tables = page.extract_tables()
for t_idx, table in enumerate(tables):
table_md = table_to_markdown(table)
docs.append(Document(
content=table_md,
meta={"page": page_num, "type": "table", "table_index": t_idx}
))
# Extract text with font/size info for heading detection
chars = page.chars
sections = detect_sections_from_fonts(chars)
for section in sections:
docs.append(Document(
content=section["text"],
meta={"page": page_num, "heading": section["heading"], "level": section["level"]}
))
return docsTradeoffs: Maximum retrieval precision for structured docs. Chunks align with how humans navigate the content. But: extraction is fragile — PDF structure is not semantic structure. Header detection fails on scanned docs, weird fonts, or generated PDFs. Maintenance burden is high.
When to use: Technical documentation, API specs, regulatory documents, any corpus where section boundaries are meaningful and extractable.
Hybrid approaches
Production systems often combine strategies. A common pattern: structure-aware first pass, then semantic or fixed-size fallback for unstructured sections.
@component
class HybridDocumentSplitter:
def __init__(self):
self.markdown_splitter = MarkdownDocumentSplitter(split_level=2)
self.fallback_splitter = DocumentSplitter(split_by="token", split_length=300, split_overlap=50)
self.semantic_splitter = SemanticDocumentSplitter(threshold=0.7)
@component.output_types(documents=List[Document])
def run(self, documents: List[Document]):
all_chunks = []
for doc in documents:
# Try structure-aware first
if doc.meta.get("format") == "markdown" or self._has_headers(doc.content):
result = self.markdown_splitter.run([doc])
chunks = result["documents"]
# Further split oversized chunks
final_chunks = []
for chunk in chunks:
if self._estimate_tokens(chunk.content) > 400:
sub = self.semantic_splitter.run([chunk])
final_chunks.extend(sub["documents"])
else:
final_chunks.append(chunk)
all_chunks.extend(final_chunks)
else:
# Unstructured: semantic with fixed fallback
result = self.semantic_splitter.run([doc])
all_chunks.extend(result["documents"])
return {"documents": all_chunks}
def _has_headers(self, text: str) -> bool:
return bool(re.search(r'^#{1,3}\s', text, re.MULTILINE))
def _estimate_tokens(self, text: str) -> int:
return len(text) // 4 # rough heuristicThis gives you header-aligned chunks where possible, semantic coherence elsewhere, and a hard token ceiling everywhere.
Evaluating your chunking strategy
Don’t guess. Measure. Two evaluation axes: retrieval metrics and end-to-end answer quality.
Retrieval evaluation
Build a small labeled set: (query, relevant_doc_ids). Compare chunking strategies on recall@k and nDCG.
from haystack import Pipeline
from haystack.components.retrievers import InMemoryEmbeddingRetriever
from haystack.components.embedders import SentenceTransformersTextEmbedder
from haystack.evaluation import Evaluator
from haystack.evaluation.metrics import Recall, NDCG
def evaluate_chunking_strategy(documents, queries, relevant_docs, document_store):
# Index with current strategy
indexing_pipeline = build_indexing_pipeline(document_store)
indexing_pipeline.run({"documents": documents})
# Retrieval pipeline
retrieval = Pipeline()
retrieval.add_component("embedder", SentenceTransformersTextEmbedder())
retrieval.add_component("retriever", InMemoryEmbeddingRetriever(document_store, top_k=10))
retrieval.connect("embedder.embedding", "retriever.query_embedding")
# Evaluate
evaluator = Evaluator(
pipeline=retrieval,
metrics=[Recall(k=5), NDCG(k=5)],
inputs=[{"query": q, "relevant_documents": rel} for q, rel in zip(queries, relevant_docs)]
)
return evaluator.run()Run this for each strategy. You’ll often find semantic chunking wins on recall@5 for conceptual queries, while structure-aware wins on nDCG for navigational queries (“show me the authentication section”).
End-to-end RAG evaluation
Retrieval metrics don’t capture generation quality. Use a small golden set of (query, expected_answer) pairs and judge with an LLM evaluator.
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import PromptBuilder
rag_pipeline = Pipeline()
rag_pipeline.add_component("embedder", SentenceTransformersTextEmbedder())
rag_pipeline.add_component("retriever", InMemoryEmbeddingRetriever(document_store, top_k=5))
rag_pipeline.add_component("prompt", PromptBuilder(template=RAG_TEMPLATE))
rag_pipeline.add_component("generator", OpenAIGenerator(model="gpt-4o-mini"))
rag_pipeline.connect("embedder.embedding", "retriever.query_embedding")
rag_pipeline.connect("retriever.documents", "prompt.documents")
rag_pipeline.connect("prompt.prompt", "generator.prompt")
# LLM-as-judge evaluation
judge_prompt = """
Score the answer 1-5 on: accuracy, completeness, conciseness.
Query: {{query}}
Expected: {{expected}}
Actual: {{actual}}
Return JSON: {"accuracy": int, "completeness": int, "conciseness": int}
"""
judge = OpenAIGenerator(model="gpt-4o", generation_kwargs={"response_format": {"type": "json_object"}})Compare average scores across chunking strategies. This catches cases where retrieval looks good but chunks are too fragmented for coherent synthesis.
Common pitfalls and tradeoffs
| Pitfall | Symptom | Fix |
|---|---|---|
| No overlap | Answers cut off mid-thought | Always use 10-20% overlap |
| Fixed token count on word-split | Chunks exceed model context | Use split_by="token" with the retriever’s tokenizer |
| Semantic threshold too high | Over-fragmentation, thousands of tiny chunks | Set min_chunk_sentences=3-5, tune threshold on dev set |
| Ignoring metadata | Can’t filter by section, source, date | Propagate doc.meta through every splitter; add chunk_id, parent_id |
| Chunking tables as text | Unretrievable tabular data | Extract tables separately, embed as markdown + structured JSON |
| One strategy for all docs | Poor recall on heterogeneous corpus | Route by mime-type / structure detection to specialized splitters |
Latency budget: Semantic splitting adds ~50-200ms per document at index time (embedding sentences). For high-volume pipelines, pre-compute embeddings offline or use a lighter model (MiniLM) for splitting, heavier for retrieval.
Index size: Overlap and semantic fragmentation increase document count. A 10K doc corpus at 250 tokens/50 overlap → ~60K chunks. At 512 tokens/100 overlap → ~25K chunks. Plan your vector store capacity accordingly.
Re-chunking is expensive: Changing strategy means re-embedding the full corpus. Design for this: version your chunking config, store chunking_version in document metadata, and build a re-indexing pipeline from day one.
Choosing a starting point
- Prototype fast: Fixed token chunking (256/32) with your retriever’s tokenizer. Baseline in 15 minutes.
- Diagnose failures: Run retrieval eval. Look at false negatives — are they split across chunks? Truncated? Buried in noise?
- Match strategy to failure mode:
- Split concepts → semantic
- Navigational queries fail → structure-aware
- Tables/lists broken → specialized extractors
- Hybridize: Route by document type. Most corpora are mixed.
- Automate eval: CI gate on recall@5 and LLM-judge score. Prevent regressions when you tweak thresholds.
The best chunking strategy is the one you can measure, explain, and change. Start simple enough to debug when the 2 AM page comes in.