Choosing the right chunk size for RAG is one of the highest-leverage decisions you’ll make when building a retrieval system. Too small and you lose context; too large and you dilute relevance and blow your token budget. This guide walks through a repeatable process to find the sweet spot for your data and queries, with code you can run today.
Step 1: Understand the tradeoffs before you write code
Chunk size directly controls three things: how much context travels with each embedding, how many chunks you retrieve per query, and how much of your context window gets consumed by retrieval overhead. A 256-token chunk might fit 20 results in a 4k window but miss the answer because the relevant sentence sits across a boundary. A 2048-token chunk preserves context but returns fewer distinct passages and wastes tokens on filler.
The embedding model also matters. Models trained with a specific context window (text-embedding-3-small at 8k, voyage-3 at 32k, bge-large-en-v1.5 at 512) produce degraded representations when you feed them chunks significantly larger or smaller than what they saw during training. Check the model card.
Start with these heuristics:
- General web/docs: 512–1024 tokens
- Code: 256–512 tokens (functions are naturally smaller)
- Legal/regulatory: 1024–2048 tokens (clauses reference each other)
- Conversation logs: 256–512 tokens (turns are short)
But heuristics are just starting points. The rest of this guide shows how to validate and tune.
Step 2: Build a reproducible chunking pipeline
Don’t hand-roll splitting logic. Use a library that handles token counting correctly and gives you deterministic, configurable splits. LangChain’s RecursiveCharacterTextSplitter and LlamaIndex’s SentenceSplitter are the two most common choices. Both count tokens with tiktoken (OpenAI) or the appropriate tokenizer for other models.
# chunking_pipeline.py
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
import tiktoken
def get_splitter(chunk_size: int, chunk_overlap: int, model_name: str = "text-embedding-3-small"):
encoding = tiktoken.encoding_for_model(model_name)
return RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
length_function=lambda text: len(encoding.encode(text)),
separators=["\n\n", "\n", ". ", " ", ""],
)
def chunk_documents(docs, chunk_size: int, chunk_overlap: int):
splitter = get_splitter(chunk_size, chunk_overlap)
return splitter.split_documents(docs)Key parameters:
chunk_size: target tokens per chunk (not characters)chunk_overlap: tokens shared between adjacent chunks. Start at 10–20% of chunk size. Overlap prevents boundary losses but increases index size and retrieval latency linearly.
Save the chunk configuration alongside your index metadata. You’ll need it for reproduction and for any re-indexing jobs.
Step 3: Create a labeled evaluation set
You cannot tune what you cannot measure. Build a small (50–200) set of realistic queries with ground-truth answers and the specific document spans that contain each answer. This is the most tedious part of the process and the most valuable.
Format each example as:
{
"query": "What is the refund policy for annual subscriptions?",
"answer": "Annual subscriptions are refundable within 30 days on a pro-rated basis.",
"source_doc_ids": ["doc-12", "doc-12"],
"source_spans": [
{"doc_id": "doc-12", "start_char": 3400, "end_char": 3580},
{"doc_id": "doc-12", "start_char": 4100, "end_char": 4250}
]
}source_spans are the exact character ranges in the raw documents that a perfect retriever should surface. If you don’t have spans, at minimum record which document IDs contain the answer. Without this, you’re guessing.
Pull queries from real user logs if you have them. If not, write them yourself covering:
- Fact lookup (“What is X?”)
- Multi-hop (“How do X and Y relate?”)
- Negation (“Does the policy forbid Z?”)
- Ambiguous/underspecified (“Tell me about pricing”)
Step 4: Run a retrieval sweep across chunk sizes
Now automate the experiment. For each candidate chunk size, build an index, run your eval queries, and compute retrieval metrics. Use a vector store that supports fast re-indexing (Chroma, Qdrant, Pinecone, Weaviate all work).
# eval_sweep.py
import json
from pathlib import Path
from chunking_pipeline import chunk_documents, get_splitter
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from tqdm import tqdm
CHUNK_SIZES = [256, 384, 512, 768, 1024, 1536, 2048]
OVERLAP_RATIO = 0.15
EVAL_SET = "eval_set.jsonl"
DOCS_PATH = "raw_docs/"
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
def load_docs():
# Your document loading logic here
pass
def load_eval():
with open(EVAL_SET) as f:
return [json.loads(line) for line in f]
def recall_at_k(retrieved_ids, relevant_ids, k):
return len(set(retrieved_ids[:k]) & set(relevant_ids)) / len(relevant_ids)
def mrr_at_k(retrieved_ids, relevant_ids, k):
for i, doc_id in enumerate(retrieved_ids[:k]):
if doc_id in relevant_ids:
return 1.0 / (i + 1)
return 0.0
def run_sweep():
docs = load_docs()
eval_set = load_eval()
results = []
for chunk_size in CHUNK_SIZES:
overlap = int(chunk_size * OVERLAP_RATIO)
chunks = chunk_documents(docs, chunk_size, overlap)
# Build fresh index per chunk size
persist_dir = f"./chroma_index_{chunk_size}"
vectorstore = Chroma.from_documents(
chunks, embeddings, persist_directory=persist_dir
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 10})
recalls, mrrs = [], []
for ex in tqdm(eval_set, desc=f"chunk_size={chunk_size}"):
retrieved = retriever.invoke(ex["query"])
retrieved_ids = [d.metadata["doc_id"] for d in retrieved]
relevant_ids = ex["source_doc_ids"]
recalls.append(recall_at_k(retrieved_ids, relevant_ids, k=5))
mrrs.append(mrr_at_k(retrieved_ids, relevant_ids, k=5))
results.append({
"chunk_size": chunk_size,
"overlap": overlap,
"num_chunks": len(chunks),
"recall@5": sum(recalls) / len(recalls),
"mrr@5": sum(mrrs) / len(mrrs),
})
with open("sweep_results.json", "w") as f:
json.dump(results, f, indent=2)
if __name__ == "__main__":
run_sweep()Run this overnight. Plot recall@5 and mrr@5 vs. chunk size. You’ll typically see a curve that peaks somewhere in the 512–1536 range and drops off on either side.
Verification: The sweep completes and produces sweep_results.json with metrics for each chunk size. Plot the results — you should see a clear peak, not a flat line. If it’s flat, your eval set is too easy or too small.
Step 5: Inspect failure cases at the peak
Metrics alone hide pathology. Pull the queries where the best chunk size still fails (recall@5 = 0) and manually inspect what happened.
# analyze_failures.py
import json
from chunking_pipeline import chunk_documents
def inspect_failures(best_chunk_size: int, eval_set_path: str, docs):
overlap = int(best_chunk_size * 0.15)
chunks = chunk_documents(docs, best_chunk_size, overlap)
# Build a lookup: doc_id -> list of (chunk_text, start_char, end_char)
from collections import defaultdict
chunk_map = defaultdict(list)
for chunk in chunks:
doc_id = chunk.metadata["doc_id"]
chunk_map[doc_id].append({
"text": chunk.page_content,
"start": chunk.metadata.get("start_index", 0),
"end": chunk.metadata.get("end_index", 0),
})
with open(eval_set_path) as f:
eval_set = [json.loads(line) for line in f]
for ex in eval_set:
# Check if any ground-truth span falls inside a single chunk
covered = False
for span in ex["source_spans"]:
doc_id = span["doc_id"]
for chunk in chunk_map[doc_id]:
if chunk["start"] <= span["start_char"] and chunk["end"] >= span["end_char"]:
covered = True
break
if covered:
break
if not covered:
print(f"QUERY: {ex['query']}")
print(f" Ground truth spans: {ex['source_spans']}")
print(f" Chunks for doc {ex['source_doc_ids'][0]}:")
for chunk in chunk_map[ex['source_doc_ids'][0]][:3]:
print(f" [{chunk['start']}-{chunk['end']}] {chunk['text'][:120]}...")
print()Common failure patterns:
- Boundary splits: The answer straddles two chunks. Increase overlap or use semantic chunking (see Step 6).
- Chunk too small: The answer requires two paragraphs that never appear together. Increase chunk size.
- Chunk too large: The relevant sentence is buried in 2k tokens of boilerplate. Decrease chunk size or add metadata filtering.
- Wrong granularity: Your docs have tables, code blocks, or nested lists that the recursive splitter mangles. Add structure-aware splitting.
Verification: You can explain every zero-recall query in terms of a specific chunking artifact, not “the model didn’t understand.”
Step 6: Try semantic chunking if recursive splitting plateaus
Recursive splitting is structure-agnostic. If your documents have clear semantic boundaries (sections, functions, Q&A pairs), a semantic chunker often beats the best fixed-size recursive split by 5–15% recall.
# semantic_chunking.py
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
def semantic_chunker(embedding_model="text-embedding-3-small", breakpoint_threshold_type="percentile", breakpoint_threshold_amount=95):
embeddings = OpenAIEmbeddings(model=embedding_model)
return SemanticChunker(
embeddings,
breakpoint_threshold_type=breakpoint_threshold_type,
breakpoint_threshold_amount=breakpoint_threshold_amount,
)SemanticChunker embeds each sentence, computes cosine similarity between adjacent sentences, and splits where similarity drops below a threshold. The percentile mode with 95 means “split at the 5% most dissimilar adjacent pairs.”
Run the same sweep (Step 4) with semantic chunking. Compare the best semantic result against the best fixed-size result. If semantic wins, adopt it — but note that it’s slower to index (embedding every sentence) and less deterministic (embedding model updates change boundaries).
Verification: Semantic chunking beats your best fixed-size baseline on recall@5 by a meaningful margin (>3-5%) on your eval set.
Step 7: Validate end-to-end with generation quality
Retrieval metrics are proxies. The real test is whether the LLM produces correct answers when given the retrieved chunks. Run a small generation eval using your peak chunk size.
# generation_eval.py
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
PROMPT = ChatPromptTemplate.from_template("""Answer the question using only the provided context. If the context doesn't contain the answer, say "I don't know."
Context:
{context}
Question: {question}
Answer:""")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
chain = PROMPT | llm | StrOutputParser()
def evaluate_generation(eval_set, retriever, max_examples=50):
correct, total = 0, 0
for ex in eval_set[:max_examples]:
retrieved = retriever.invoke(ex["query"])
context = "\n\n".join(d.page_content for d in retrieved)
prediction = chain.invoke({"context": context, "question": ex["query"]})
# Simple semantic equivalence check - use an LLM judge in production
judge_prompt = f"""Does the predicted answer correctly answer the question based on the ground truth?
Question: {ex['query']}
Ground truth: {ex['answer']}
Predicted: {prediction}
Answer YES or NO only."""
judgment = llm.invoke(judge_prompt).content.strip().upper()
if judgment == "YES":
correct += 1
total += 1
print(f"Q: {ex['query']}")
print(f" Pred: {prediction[:100]}...")
print(f" Truth: {ex['answer'][:100]}...")
print(f" Judge: {judgment}\n")
print(f"Generation accuracy: {correct}/{total} = {correct/total:.2%}")This is expensive (LLM calls) so limit to 50–100 examples. In production, swap the manual judge for a calibrated LLM-as-judge or a fine-tuned classifier.
Verification: Generation accuracy at your chosen chunk size is within 5% of the best chunk size you tested. If a smaller chunk size has slightly lower recall but higher generation accuracy (less noise in context), prefer it.
Step 8: Lock it in and add production monitoring
Once you’ve selected a chunk size, treat it as a configuration parameter — not a hardcoded constant. Store it in your index metadata and version it with your deployment.
# index_metadata.json
{
"chunk_size": 768,
"chunk_overlap": 115,
"splitter": "RecursiveCharacterTextSplitter",
"embedding_model": "text-embedding-3-small",
"created_at": "2025-01-15T14:30:00Z",
"eval_recall_at_5": 0.87,
"eval_mrr_at_5": 0.72
}Add monitoring for retrieval quality drift. The simplest approach: log every query with its retrieved chunk IDs and the user’s implicit/explicit feedback (thumbs up/down, reformulation, click-through). Compute rolling recall@5 against a shadow eval set weekly.
-- Example: weekly recall computation from logs
WITH eval_queries AS (
SELECT query, relevant_doc_ids FROM golden_eval_set
),
retrieval_logs AS (
SELECT query, retrieved_doc_ids[0:5] AS top5
FROM retrieval_logs
WHERE timestamp > NOW() - INTERVAL '7 days'
)
SELECT
AVG(CASE WHEN array_intersect(top5, relevant_doc_ids) > 0 THEN 1.0 ELSE 0.0 END) AS recall_at_5
FROM eval_queries
JOIN retrieval_logs USING (query);Alert if recall@5 drops more than 5 percentage points from baseline. Common causes: new document types entering the corpus, embedding model updates, or upstream parsing changes.
Verification: Your monitoring dashboard shows current recall@5 within 3 points of the baseline you measured in Step 4.
Step 9: Handle mixed document types with tiered chunking
Real corpora are heterogeneous. A single chunk size rarely optimizes across API references, prose documentation, changelogs, and support tickets. Implement a routing layer that selects the chunking strategy per document type.
# tiered_chunking.py
from chunking_pipeline import chunk_documents, get_splitter
from semantic_chunking import semantic_chunker
STRATEGIES = {
"api_reference": {"method": "semantic", "params": {}},
"documentation": {"method": "fixed", "params": {"chunk_size": 1024, "overlap": 150}},
"changelog": {"method": "fixed", "params": {"chunk_size": 512, "overlap": 80}},
"support_ticket": {"method": "fixed", "params": {"chunk_size": 384, "overlap": 60}},
"code": {"method": "fixed", "params": {"chunk_size": 256, "overlap": 40}},
}
def chunk_by_type(doc):
doc_type = doc.metadata.get("doc_type", "documentation")
strategy = STRATEGIES.get(doc_type, STRATEGIES["documentation"])
if strategy["method"] == "semantic":
splitter = semantic_chunker()
else:
splitter = get_splitter(**strategy["params"])
return splitter.split_documents([doc])Tag documents at ingestion time (file extension, directory, metadata field). Re-run your eval sweep per document type if you have enough labeled examples per type. At minimum, verify that the tiered approach doesn’t regress overall metrics.
Verification: Tiered chunking matches or beats the single best fixed size on the full eval set, and improves recall on at least one document type by >10%.
Step 10: Re-evaluate when the embedding model changes
Embedding models are not interchangeable. If you switch from text-embedding-3-small to voyage-3-large or bge-m3, the optimal chunk size can shift by 2x. The model’s training context window, pooling strategy, and fine-tuning data all affect how it represents long vs. short texts.
When you change embedding models:
- Re-run Step 4 (the sweep) with the new model
- Re-run Step 7 (generation eval) — retrieval metrics can improve while generation quality degrades if the new model retrieves different but semantically similar passages that confuse the LLM
- Update your index metadata with the new model name and chunk size
Budget for a full re-index. It’s not optional.
Verification: New model + new chunk size beats old model + old chunk size on generation eval (Step 7), not just retrieval metrics.
The chunk size for RAG that works today won’t work forever. Corpus composition shifts, embedding models improve, and user query patterns evolve. Treat chunking as a tunable hyperparameter with a regular re-evaluation cadence — quarterly at minimum, monthly if you’re actively adding document types. The evaluation infrastructure you built in Steps 3–4 is the real asset; the chunk size is just the current output.