Most retrieval-augmented generation failures trace back to a weak chunking strategy token limits RAG plan rather than the model itself. If your chunks overflow the embedding model’s max input or starve the LLM of room to reason, retrieval returns fragments that look relevant but lose the surrounding context.
Why chunk size is a token budget problem
Chunking is not about characters or pages. It is about tokens: the atomic unit your embedding model and completion model both consume. Every embedding model has a hard input cap (OpenAI’s text-embedding-3-small accepts 8191 tokens, older ada-002 capped at 8191 as well, while some open-weight models sit at 512). Exceed it and the SDK silently truncates or errors. On the generation side, a 128k context window does not mean you should fill it with 100k of retrieved text—attention degrades and cost scales linearly.
A defensible chunking strategy token limits RAG approach starts by treating the document as a token stream and the pipeline as a fixed budget split across embed, retrieve, and generate phases.
Step 1: Profile documents and model constraints
Before writing a splitter, measure. Pull a representative sample of your corpus and tokenize it. Use the same tokenizer the embedding model uses—for OpenAI models that is cl100k_base via tiktoken.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
def token_len(text: str) -> int:
return len(enc.encode(text))
sample = open("contract.pdf.txt").read()
print(f"doc tokens: {token_len(sample)}")Record three numbers: median doc length, p95 doc length, and the longest single logical section (e.g., a clause or function). Compare against your embedding model max tokens and your completion model’s context window minus expected output tokens.
Step 2: Choose chunk size and overlap from query granularity
Queries in RAG are usually short and target a specific fact. If a user asks “What is the termination penalty?” the answer likely lives in a 200–400 token span. Chunks of 1024 tokens with 10% overlap work for prose; code or tables often need smaller, structure-aware chunks.
Rule of thumb:
- Prose: 256–512 tokens, 15% overlap
- Technical docs: 128–256 tokens, 20% overlap
- Source code: per-function or 64–128 tokens, no blind character split
Overlap prevents boundary cuts but costs storage and can cause duplicate hits. Cap overlap so retrieved set deduplication is trivial.
CHUNK_TOKENS = 384
OVERLAP_TOKENS = 64
def chunk_by_tokens(text, chunk_t=CHUNK_TOKENS, overlap_t=OVERLAP_TOKENS):
tokens = enc.encode(text)
step = chunk_t - overlap_t
for i in range(0, len(tokens), step):
yield enc.decode(tokens[i:i+chunk_t])Step 3: Split on structure, not just characters
Character splitters that cut every N tokens ignore headings, list items, and code fences. A sentence mid-way through a numbered list loses its referent. Prefer recursive splitting that respects separators in priority order.
SEPS = ["\n## ", "\n### ", "\n\n", "\n", ". ", " "]
def recursive_split(text, max_tok):
if token_len(text) <= max_tok:
return [text]
for sep in SEPS:
if sep in text:
left, right = text.split(sep, 1)
# reattach sep to right for context
right = sep + right
return recursive_split(left, max_tok) + recursive_split(right, max_tok)
# fallback: hard token cut
return list(chunk_by_tokens(text, max_tok, 0))For Markdown, keep heading ancestry in each chunk: prepend the H1/H2 path so the embedding carries section context. This lifts recall without increasing token count.
Step 4: Enforce limits at ingest and query time
Never assume upstream data is clean. At ingest, reject or split any block exceeding the embedding limit. At query time, compute how many chunks fit in the completion context.
SYS_TOK = 120
QUERY_TOK = 64
OUTPUT_RESERVE = 1024
MODEL_CTX = 32768
def max_retrieval_tokens(model_ctx=MODEL_CTX):
return model_ctx - SYS_TOK - QUERY_TOK - OUTPUT_RESERVE
def chunks_to_fetch(chunk_tok, k):
budget = max_retrieval_tokens()
return min(k, budget // (chunk_tok + 8)) # 8 tok overhead per chunk metadataIf you retrieve with a vector DB that returns scores, apply this k cap before sending to the model.
Step 5: Map chunk tokens to a retrieval and generation budget
A common pitfall is setting k=10 because a tutorial did. With 512-token chunks that is 5k tokens—fine for a 32k model, dangerous for an 8k one. Build the budget explicitly:
{
"embedding_model_max_tokens": 8191,
"chunk_tokens": 384,
"overlap_tokens": 64,
"index_overhead_tokens_per_chunk": 8,
"completion_model_context": 32768,
"system_prompt_tokens": 120,
"query_tokens": 64,
"output_reserve_tokens": 1024,
"max_k": 48
}Recalculate max_k whenever you swap models. This is where an inference gateway helps: if you route through an OpenAI-compatible endpoint like n4n.ai that honors client routing directives and forwards provider cache-control hints, you can change the completion model without rewriting chunk-size logic in every service.
Common pitfalls in chunking strategy token limits RAG
Silent truncation at embed time. The embedding API drops tokens beyond its limit. Your index quietly misses the second half of long clauses. Always pre-split.
Overlap inflation. 30% overlap on 10k chunks means you store 30% more vectors and pay 30% more query compute for marginal recall gain. Measure recall@k on a labeled set; tune overlap down until it drops.
Ignoring metadata tokens. If you inject source: file.pdf page 12 per chunk, that counts. A 384-token chunk plus 32 tokens metadata becomes 416. Update the budget.
Retrieving by fixed k, not by token sum. Two chunks of 200 tokens and one of 900 should not be treated identically. Sort by score, then fill the token budget greedily.
No separation of embed vs completion tokenizer. Using cl100k_base for a Cohere embed model is wrong; mismatch causes off-by-2x token counts. Check the model card.
Structure-aware chunking for code RAG
Code does not obey prose rules. A function is the unit. Use a parser (tree-sitter or language server) to extract top-level definitions, then chunk only if a single definition exceeds the limit.
# pseudo: tree-sitter extraction
from tree_sitter import Parser, Language
parser = Parser()
parser.set_language(Language("build/python.so", "python"))
def extract_functions(source):
tree = parser.parse(bytes(source, "utf8"))
# walk tree for function_definition nodes, return node.text
...If a function is 1200 tokens, fall back to splitting by logical blocks (loop, branch) with overlap that includes the function signature.
A worked end-to-end config
Below is a minimal ingest pipeline that respects token limits and emits overlap with heading context.
import re, tiktoken
enc = tiktoken.get_encoding("cl100k_base")
def add_heading_context(text, chunk):
heads = re.findall(r"(?m)^(#{1,2})\s+(.+)$", text)
prefix = " > ".join(h[1] for h in heads[:2])
return f"{prefix}\n\n{chunk}" if prefix else chunk
def ingest(doc_text, chunk_t=384, overlap_t=64):
chunks = []
for sec in recursive_split(doc_text, chunk_t):
toks = enc.encode(sec)
for i in range(0, len(toks), chunk_t - overlap_t):
piece = enc.decode(toks[i:i+chunk_t])
chunks.append(add_heading_context(doc_text, piece))
return chunks # each under embed limit, ready for vector storeAt query time, compute k from the live completion model context and fetch that many. Log the actual token sum per request to catch drift.
What to measure next
Track three metrics from day one: mean retrieved token sum, recall@k on a 50-question gold set, and p95 time-to-first-token with full context. When recall plateaus but latency climbs, reduce chunk_t and increase k instead of padding chunks. The right chunking strategy token limits RAG tuning is iterative, not a one-time config.
If your pipeline spans multiple providers, keep the token math in one module and pass model identifiers explicitly. That isolates the only part of the system that must change when a new model with a different context window appears.