Chunk overlap in LangChain: how much is too much

Getting langchain chunk overlap tuning wrong silently degrades retrieval or inflates your token bill. The default 200/20 split in many tutorials is a starting point, not a target; the right numbers depend on your document structure and embedding model.

Why overlap exists

A chunker slices documents into pieces small enough for an embedding model and a retriever. Without overlap, any sentence that straddles a boundary gets cut, and the embedding for each half loses the missing context. Overlap duplicates a slice of the previous chunk at the start of the next one, so boundary-spanning ideas stay intact.

The cost is real: every overlapped token is stored, embedded, and possibly re-sent to a model twice. Treat overlap as insurance, not free headroom.

Step 1: Profile your documents

Before setting any numbers, measure what you are splitting. Load a representative sample and look at paragraph and sentence lengths.

import statistics

def profile(texts):
    paras = [t for doc in texts for t in doc.split("\n\n")]
    lengths = [len(p.split()) for p in paras]
    print(f"paras: {len(lengths)}")
    print(f"median words/para: {statistics.median(lengths)}")
    print(f"p95 words/para: {sorted(lengths)[int(0.95*len(lengths))]}")

profile(raw_texts)

If your median paragraph is 80 words and p95 is 300, a chunk_size of 200 words will split most paragraphs. That pushes you toward larger chunks or a separator strategy that respects paragraphs.

Step 2: Pick a chunk_size anchored to the embedder

Most modern embedding models handle 512–8192 tokens, but retrieval quality peaks when chunks map to a single coherent idea. For text-embedding-3-small or similar, 256–512 tokens is a sane range.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    length_function=len,
    separators=["\n\n", "\n", ". ", " "]
)

The separators list tells LangChain to break at paragraph boundaries first, then lines, then sentences, then words. This alone fixes a lot of bad splits that pure character counting causes.

Step 3: Set overlap as a fraction of chunk_size

A practical rule for langchain chunk overlap tuning: start at 10–20% of chunk_size. For 512, that is 51–102 tokens. Code and technical docs often need 25–40% because a function signature or import block at the tail of one chunk should appear in the next.

chunk_size = 512
overlap_ratio = 0.15
chunk_overlap = int(chunk_size * overlap_ratio)  # 76

Avoid hard-coding 20 because it stops making sense the moment you change chunk_size.

Step 4: Validate with retrieval, not vibes

Build a small golden set: 20–50 questions with known answer spans in the source docs. Split the docs, embed, retrieve top-k, and check if the span appears.

from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings

docs = splitter.split_documents(raw_docs)
vs = FAISS.from_documents(docs, OpenAIEmbeddings())
retriever = vs.as_retriever(search_kwargs={"k": 4})

hit = 0
for q, answer_span in golden:
    ctx = " ".join(d.page_content for d in retriever.get_relevant_documents(q))
    if answer_span in ctx:
        hit += 1
print(f"recall@4: {hit/len(golden):.2f}")

Run this at overlap 0, 10%, 20%, 30%. Recall usually climbs then plateaus. Past the plateau you are paying tokens for no gain.

Step 5: Quantify the token tax

Overlap increases chunk count. Approximate total embedded tokens:

total_tokens ≈ source_tokens * (1 + overlap / (chunk_size - overlap))

At 512/64, that is ~1.14x. At 512/256, it is ~2x. If you later send those chunks to a generator, the tax repeats on every retrieved context.

If you route embedding or completion traffic through a gateway, per-token usage metering makes the cost visible per pipeline. n4n.ai exposes exactly that across 240+ models behind one endpoint, so a regressions in langchain chunk overlap tuning shows up as a line-item bump rather than a mystery invoice.

Common pitfalls

Overlap exceeds 50% of chunk_size

You create near-duplicate chunks. Vector stores and rerankers treat them as independent evidence, which biases scores and wastes memory. Stop at 40% unless you have a specific boundary-heavy format.

One overlap for every document type

PDF research papers, Slack exports, and Python source have different boundary structures. Configure splitters per loader:

code_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800, chunk_overlap=200, separators=["\nclass ", "\ndef ", "\n", " "]
)
text_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)

Ignoring length_function mismatch

LangChain defaults to len (character count). If your embedding counts tokens, a 512-character chunk may be 120 tokens or 400 tokens depending on language. Use a tokenizer-backed length function for non-English or code-heavy corpora.

Treating overlap as context window padding

Overlap is not a substitute for a larger chunk. If you need more surrounding text, raise chunk_size first, then adjust overlap.

A config pattern that survives contact with production

Externalize the values and default to the profiling-based choices:

import os

CHUNK_SIZE = int(os.getenv("CHUNK_SIZE", "512"))
CHUNK_OVERLAP = int(os.getenv("CHUNK_OVERLAP", "64"))
SEPARATORS = ["\n\n", "\n", ". ", " ", ""]

def make_splitter():
    return RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
        separators=SEPARATORS,
        length_function=len,
    )

This lets you run A/B retrieval tests by flipping env vars in CI without code changes.

When to retune

Revisit langchain chunk overlap tuning when any of these happen:

• You swap embedding models (token limits and semantic granularity change).
• Document distribution shifts (e.g., you add long PDFs to a previously chat-only corpus).
• Retrieval recall drops below your threshold after a content update.
• Your token cost per query creeps up without query-volume growth.

Set a quarterly reminder to re-run the golden-set evaluation. The numbers that were right at 10k documents are often wrong at 10M.

Tradeoff summary

Small overlap: fewer tokens, faster ingest, but higher risk of broken context at boundaries. Large overlap: safer retrieval for boundary-spanning facts, but quadratic-ish storage growth and potential duplicate-chunk noise. The job is to find the knee of the recall curve, then shave overlap until the metric moves.

Most teams land between 10% and 25% for prose and 30%–40% for code. Start there, measure, and let the golden set dictate the final value.