Chunking strategies for LlamaIndex document ingestion

Bad chunking silently ruins RAG systems. This llamaindex chunking strategies tutorial walks through a concrete ordering of splitters and parameters you can apply to real document ingestion pipelines, starting from the defaults and moving to structure-aware and semantic approaches.

Why chunking decides retrieval quality

The embedding model sees each chunk as a single fixed-length vector. If a chunk mixes two unrelated topics, the vector averages them and retrieves poorly for both. If a chunk is too small, it loses co-reference and context. The splitter is the first and highest-leverage knob in any LlamaIndex ingestion job.

Retrieval accuracy is downstream of chunk boundaries, not just the embed model. You can swap models later; re-chunking means re-embedding everything. In this llamaindex chunking strategies tutorial we focus on splitters that are stable, measurable, and cheap to iterate on.

1. Baseline: TokenTextSplitter

LlamaIndex’s TokenTextSplitter is the safest starting point because it aligns with token limits of your embed model and LLM. Use it to enforce a hard ceiling that fits your vector store and context window.

from llama_index.core import Document
from llama_index.core.node_parser import TokenTextSplitter

doc = Document(text=open("contract.txt").read())
splitter = TokenTextSplitter(chunk_size=512, chunk_overlap=64)
nodes = splitter.get_nodes_from_documents([doc])

chunk_size is in tokens, not characters. Set it to roughly half the embed model’s max context if you plan to pack metadata. Overlap of 10–15% mitigates boundary cuts.

Pitfall: default chunk_size=1024 with OpenAI text-embedding-3-small (8192-token limit) is fine, but if you later switch to a local BGE model with 512-token limit, your chunks silently truncate. Always bind chunk size to the embed model, not the LLM. Verify with tiktoken:

import tiktoken
enc = tiktoken.encoding_for_model("text-embedding-3-small")
print(len(enc.encode(nodes[0].get_content())))

2. Preserve structure with MarkdownNodeParser

Prose isn’t the only input. Engineering docs, READMEs, and Notion exports are Markdown. MarkdownNodeParser keeps headings as node metadata and nests sections instead of slicing mid-heading.

from llama_index.core.node_parser import MarkdownNodeParser

parser = MarkdownNodeParser()
md_nodes = parser.get_nodes_from_documents([doc])

Each node gets node.metadata["section_title"] and parent links. This preserves hierarchy so a query about “auth middleware” retrieves the exact subsection, not a random slice that happened to contain the word. You can inspect the tree:

for n in md_nodes:
    print(n.metadata.get("section_title"), "->", n.relationships.get("parent"))

Tradeoff: it assumes well-formed Markdown. Feeding it HTML or PDF text extracts produces one giant node. Clean the source first with a proper extractor.

3. SentenceSplitter for continuous prose

For books, transcripts, or support tickets, SentenceSplitter respects sentence boundaries and avoids cutting mid-sentence.

from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(
    chunk_size=1024,
    chunk_overlap=128,
    paragraph_separator="\n\n",
    secondary_chunking_regex=r"(?<=\. )",
)
nodes = splitter.get_nodes_from_documents([doc])

It uses a regex sentence tokenizer. The secondary_chunking_regex lets you break long sentences at semicolons if needed. For code or logs, set paragraph_separator to newline and lower chunk_size to 256.

Pitfall: sentence splitters can produce many tiny nodes for bullet-point lists. If your source is list-heavy, prefer the markdown parser or a custom regex.

4. Overlap, but measure it

Overlap reduces context loss at boundaries. But every overlapping token is extra embedded vectors and extra storage. At 512/64 you embed ~12% duplicate tokens. At 512/256 you embed 50% duplicates.

Set overlap based on typical sentence length in your corpus. For English technical writing, 64–128 tokens is enough. For code, use line-based overlap instead. Log the duplication ratio:

total_tokens = sum(len(enc.encode(n.get_content())) for n in nodes)
unique_tokens = len(enc.encode(" ".join(n.get_content() for n in nodes)))
print(f"duplicate ratio: {1 - unique_tokens/total_tokens:.1%}")

5. Attach metadata and relationships

A chunk without metadata is just text. LlamaIndex nodes support metadata and relationships. Stamp source file, page, and timestamp.

for n in nodes:
    n.metadata["source"] = "contract.txt"
    n.metadata["ingested_at"] = "2024-05-01"
    n.metadata["pipeline_version"] = "v1.2"
    n.relationships["parent"] = doc.as_related_node_info()

This lets you filter at query time (vector_store_query_mode with metadata filters) and trace answers back to the source. Skip this and you’ll regret it during the first audit.

6. Semantic chunking when the budget allows

If you have a reliable embedding endpoint, SemanticSplitterNodeParser breaks on meaning shifts rather than fixed sizes. It embeds sentences, computes cosine distance, and splits at percentile breakpoints.

from llama_index.core.node_parser import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding(model="text-embedding-3-small")
sem_splitter = SemanticSplitterNodeParser(
    buffer_size=1,
    breakpoint_percentile_threshold=95,
    embed_model=embed_model,
)
sem_nodes = sem_splitter.get_nodes_from_documents([doc])

buffer_size=1 means compare adjacent sentences; breakpoint_percentile_threshold=95 splits when distance is in the top 5% of the document. This yields chunks that match query intent better for heterogeneous docs. Cost: you embed every sentence twice (once for split, once for final node). For a 100k-document corpus, that’s a notable bill.

If you use an inference gateway like n4n.ai, the per-token metering makes this cost visible per ingestion run, so you can cap semantic splitting to high-value document types only.

Tradeoff: semantic splitting is slow and non-deterministic across embed model versions. Pin the embed model.

7. Evaluate before shipping

Chunking is a hypothesis. Test it with retrieval eval. Build a small set of 20 questions with known answers and measure hit rate@5.

from llama_index.core.evaluation import RetrieverEvaluator

retriever = VectorStoreIndex(nodes).as_retriever(similarity_top_k=5)
evaluator = RetrieverEvaluator.from_metric("hit_rate")
queries = {"q1": "What is the notice period?", "q2": "How is API rate limit enforced?"}
results = evaluator.evaluate(retriever, queries)
print(results)

If hit rate is below 0.8, adjust chunk_size or switch splitter. Don’t tune the prompt before tuning chunks.

Common pitfalls and tradeoffs

• Oversized chunks: They pad the vector with noise. A 2048-token chunk about “billing” that also contains “API limits” will retrieve for both weakly.
• Undersized chunks: A 128-token chunk loses the subject of a pronoun. “It fails” without context is unretrievable.
• Ignoring source format: PDFs need layout extraction (PyMuPDF or LlamaParse) before splitting. Throwing raw pdfminer text at TokenTextSplitter creates garbage nodes.
• Static overlap: Use larger overlap for dense legal text, smaller for slack messages.
• No versioning: When you change chunk size, old vectors stay in the store. Tag nodes with pipeline_version metadata and filter or rebuild.
• Semantic splitting everywhere: The embed cost scales with sentence count, not document count. Limit it.

Production checklist

The llamaindex chunking strategies tutorial steps condense into this production checklist:

1. Bind chunk_size to embed model token limit, not LLM limit.
2. Choose splitter by source: Markdown parser for docs, SentenceSplitter for prose, TokenTextSplitter for raw text.
3. Set overlap to 10–15% of chunk size; verify token count and duplication ratio.
4. Stamp source, page, pipeline_version on every node.
5. Run hit-rate eval on a 20-query set; threshold 0.8.
6. For semantic splitting, pin embed model and cap to important docs.
7. Log chunk counts and token totals per ingestion; alert on spikes.

Following the ordered path—baseline token split, structure awareness, sentence boundaries, metadata, selective semantics, eval—will get you to a defensible ingestion pipeline faster than guessing. Revisit chunking whenever you change embed models or source types.