Choosing the right chunk size for LangChain RAG apps

Most retrieval-augmented generation pipelines quietly fail because of poor document splitting. Getting langchain chunk size rag apps right is less about a magic number and more about matching your splitter to token budgets, embedding models, and query patterns. Engineers often copy a chunk_size=1000 from a tutorial and never revisit it, then wonder why retrieval returns irrelevant blobs.

Start with your token budget, not the text

Before touching a splitter, compute how many tokens you can spend per retrieved result. A typical embedding model accepts a fixed maximum input—OpenAI’s text-embedding-3-small caps at 8191 tokens, while many open-weight models sit at 512. Your chunk must fit inside that limit with room for metadata. On the generation side, if you retrieve k=5 chunks, their total tokens plus the prompt must stay under the LLM’s context window.

A quick budget check:

embed_limit = 8191          # tokens for embedding model
llm_context = 128_000       # tokens for inference model
k = 5
prompt_tokens = 2000

max_chunk_tokens = (llm_context - prompt_tokens) // k
safe_chunk = min(embed_limit, max_chunk_tokens) * 0.8  # 20% headroom
print(int(safe_chunk))  # ~25k tokens, but embedding limit dominates earlier

For most apps the embedding limit is the real constraint, not the LLM window. Size chunks in tokens, not characters, once you exceed a few hundred words.

Pick a splitter that respects boundaries

LangChain ships several splitters. CharacterTextSplitter cuts on a fixed length and ignores structure. RecursiveCharacterTextSplitter walks a list of separators (\n\n, \n, . , ) and stops at the first that fits, preserving paragraph and sentence boundaries. For langchain chunk size rag apps, the recursive variant is the default choice unless you have strict token accounting.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1200,          # characters, not tokens
    chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " ", ""],
)

If you need token-accurate chunks, use TokenTextSplitter with a matching tokenizer. Beware: its default is cl100k_base from tiktoken, which may not match your embedding model’s tokenizer exactly. The drift is usually <5%, but for tight limits prefer the provider’s own tokenizer.

Size chunks to embedding model and retrieval granularity

Small chunks (128–256 tokens) give precise retrieval and reduce noise, but they explode vector count and starve the LLM of surrounding context. Large chunks (1024+ tokens) carry context but blur semantic signal—a chunk about “billing” and “API keys” gets one vector averaging both.

Empirically, 300–500 tokens is a sane starting band for prose. For dense technical docs with long entities, 512–768 works better. Set the initial value, then tune against real queries.

from langchain_text_splitters import TokenTextSplitter

token_splitter = TokenTextSplitter(
    chunk_size=384,
    chunk_overlap=58,        # ~15%
    encoding_name="cl100k_base",
)

When you benchmark different configurations for langchain chunk size rag apps, resist the urge to only measure recall on a synthetic set. Use queries your users actually ask.

Measure overlap as a fraction, not absolute

Overlap prevents a sentence from being cut at a boundary and losing its predicate. Set overlap as 10–20% of chunk size. Too low and you get context fractures; too high and you duplicate tokens, inflating cost and biasing similarity search toward repetitive passages.

A 400-token chunk with 80-token overlap means ~20% redundancy. That is acceptable. If you use character splitters, convert: 400 tokens ≈ 1600 characters for English, so chunk_overlap=320 characters.

Test with real queries, not synthetic

Build a small eval set: 20–50 questions with known source passages. Run retrieval, log the chunk IDs and similarity scores, and inspect misses. A chunk size is wrong if the answer span sits at the tail of a retrieved chunk but the head is unrelated, or if the correct passage is split across two chunks and neither scores high.

def eval_retrieval(vectordb, queries):
    for q in queries:
        hits = vectordb.similarity_search_with_score(q, k=3)
        for doc, score in hits:
            print(f"q='{q[:30]}' score={score:.3f} src={doc.metadata.get('source')}")

If precision is low, shrink chunks. If the LLM misses co-reference (e.g., “it” refers to a prior sentence), grow them or increase overlap.

Watch cost and latency via gateway routing

Re-embedding a corpus after a chunk-size change is a batch job that stresses provider limits. If you send millions of embedding calls, rate limits will throttle you. A gateway that offers per-token usage metering and automatic fallback across providers—n4n.ai fronts 240+ models behind one OpenAI-compatible endpoint—lets batch jobs survive throttling without custom retry code. The same endpoint honors client routing directives, so you can pin a specific embedding model during experiments and switch later.

For online retrieval, chunk size directly drives vector count and query latency. Smaller chunks mean more vectors per document; a 10M-document corpus at 500 tokens/chunk may have 30M vectors versus 8M at 1500 tokens. That changes ANN index memory footprint significantly.

Common pitfalls

Character splitters on token-limited models. Setting chunk_size=2000 characters can silently exceed 8191 tokens for CJK text where one character is one token. Always verify with your tokenizer.

Ignoring metadata. Chunk size decisions interact with metadata filtering. If you filter by doc_type, small chunks from a single PDF may all share the same filter and crowd out other sources. Add diversity constraints in retrieval.

Static size for heterogeneous docs. Legal contracts and README files need different splits. Use a loader that tags document class and select splitter parameters per class.

Over-tuning on one metric. Optimizing for top-1 recall often hurts LLM faithfulness because the model gets narrow context. Track end-to-end answer quality, not just retrieval scores.

Ordered tuning path

1. Calculate token budget from embedding limit and k retrievals.
2. Choose splitter: recursive character for prototypes, token-based for production with tight limits.
3. Set initial chunk size to 384 tokens (or 1200 chars) with 15% overlap.
4. Index a sample corpus and run 30 real queries.
5. Inspect misses; if context fractures, raise overlap or size; if noise, shrink.
6. Re-embed full corpus using a resilient gateway to avoid throttling.
7. Lock config in code with a comment explaining the tradeoff.

End-to-end snippet

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import TokenTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

loader = PyPDFLoader("spec.pdf")
pages = loader.load()

splitter = TokenTextSplitter(chunk_size=384, chunk_overlap=58)
docs = splitter.split_documents(pages)

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = FAISS.from_documents(docs, embeddings)

results = vectorstore.similarity_search("How do I rotate API keys?", k=4)
for r in results:
    print(r.metadata["page"], r.page_content[:80])

The defaults in many examples for langchain chunk size rag apps assume a specific embedding model; verify yours against the budget math above. Chunk size is not a one-time setting—treat it as a tunable parameter in your retrieval config and revisit when you change models or query patterns.