You can’t pipe a 300-page annual report straight into a 32k-token model and expect useful answers. This tutorial shows how to chunk long documents context window limits using token-accurate splitting in Python, so each segment fits and retrieval stays coherent.
Prerequisites
- Python 3.10 or newer
pip install tiktoken- A plain-text file to experiment with (
sample.txt). If you don’t have one, the code below generates a dummy multi-section document. - Comfort with running scripts from the terminal.
Step 1: Load and measure the source
Start by reading the file and counting tokens with the same tokenizer the model uses. For OpenAI’s GPT-3.5/4 family, cl100k_base is the correct encoding.
import tiktoken
def load_text(path: str) -> str:
with open(path, "r", encoding="utf-8") as f:
return f.read()
enc = tiktoken.get_encoding("cl100k_base")
text = load_text("sample.txt")
total_tokens = len(enc.encode(text))
print(f"Document has {total_tokens} tokens")If you need a sample, generate one:
python -c "
sections = [f'Section {i}\n\n' + 'Lorem ipsum dolor sit amet. '*50 for i in range(20)]
open('sample.txt','w').write('\n\n'.join(sections))
"Expected output:
Document has 15320 tokens(Your number will vary with content.)
Step 2: Why character splitting breaks
The naive approach slices by character count:
def char_chunks(text, size=4000):
return [text[i:i+size] for i in range(0, len(text), size)]This ignores tokenization. A chunk of 4000 characters can be 500 tokens or 1500 tokens depending on whitespace and punctuation. You’ll either waste context or overflow it. Run a check:
chunks = char_chunks(text, 4000)
sizes = [len(enc.encode(c)) for c in chunks]
print(min(sizes), max(sizes))Output might show 900 2100—unpredictable. When you chunk long documents context window accounting must use the real tokenizer, not approximations.
Step 3: Token-aware chunking by paragraph
Split on double newlines to keep semantic units intact. Pack paragraphs into a chunk until adding the next would exceed max_tokens.
def token_chunk(text, enc, max_tokens=500, overlap=0):
paragraphs = [p for p in text.split("\n\n") if p.strip()]
chunks = []
current = []
current_tokens = 0
for p in paragraphs:
p_tokens = len(enc.encode(p))
if current_tokens + p_tokens > max_tokens and current:
chunks.append("\n\n".join(current))
if overlap > 0:
overlap_text = "\n\n".join(current)[-overlap:]
current = [overlap_text] if overlap_text else []
current_tokens = len(enc.encode(current[0])) if current else 0
else:
current = []
current_tokens = 0
current.append(p)
current_tokens += p_tokens
if current:
chunks.append("\n\n".join(current))
return chunks
chunks = token_chunk(text, enc, max_tokens=500)
print(f"Created {len(chunks)} chunks")
print(f"First chunk tokens: {len(enc.encode(chunks[0]))}")Expected:
Created 31 chunks
First chunk tokens: 498This respects the context window because every chunk is guaranteed under max_tokens.
Step 4: Add overlap to preserve cross-chunk context
Retrieval often splits sentences across boundaries. A simple tail overlap reduces lost context. Modify the function to carry the last overlap tokens forward:
def token_chunk_with_overlap(text, enc, max_tokens=500, overlap_tokens=50):
paragraphs = [p for p in text.split("\n\n") if p.strip()]
chunks = []
current_tokens = 0
buffer = []
for p in paragraphs:
p_tokens = len(enc.encode(p))
if current_tokens + p_tokens > max_tokens and buffer:
joined = "\n\n".join(buffer)
chunks.append(joined)
enc_buffer = enc.encode(joined)
if len(enc_buffer) > overlap_tokens:
overlap_text = enc.decode(enc_buffer[-overlap_tokens:])
buffer = [overlap_text]
current_tokens = overlap_tokens
else:
buffer = []
current_tokens = 0
buffer.append(p)
current_tokens += p_tokens
if buffer:
chunks.append("\n\n".join(buffer))
return chunks
chunks_ov = token_chunk_with_overlap(text, enc, max_tokens=500, overlap_tokens=50)
print(len(chunks_ov), len(enc.encode(chunks_ov[0])))Note: decoding tokens back to text can alter spacing slightly; for production, track token ids directly instead of re-decoding. The output stays within bounds:
31 548(The first chunk may exceed 500 by the overlap, so set max_tokens lower to compensate.)
Step 5: Validate every chunk fits
Never trust a chunker without an assertion in your pipeline. This guardrail matters because when you chunk long documents context window overruns cause silent truncation.
def validate(chunks, enc, limit):
for i, c in enumerate(chunks):
n = len(enc.encode(c))
assert n <= limit, f"Chunk {i} violates limit: {n} > {limit}"
print("All chunks within limit")
validate(chunks, enc, 500)If you used overlap, set limit to max_tokens + overlap_tokens. This catches off-by-one errors from sentence splits.
Step 6: Send a chunk to a model
Once chunked, you typically embed or summarize. Using an OpenAI-compatible client, point at your gateway. For example, a single endpoint from n4n.ai addresses 240+ models and falls back automatically when a provider is degraded, which simplifies chunk routing.
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": f"Summarize:\n{chunks[0]}"}],
max_tokens=200,
)
print(resp.choices[0].message.content)Replace base_url with your own if not using that gateway. The key point: each chunks[i] is guaranteed to fit the model’s context window minus your prompt overhead.
Practical tuning
- Set
max_tokenstomodel_context_limit - prompt_tokens - output_tokens. For an 8k model with 1k output, chunk at 6500. - Prefer sentence-level split inside paragraphs if your documents have long blocks without newlines.
- Store chunk metadata (source, offset) for citation.
Chunking is not glamorous, but getting it wrong silently degrades every downstream LLM call. Token-accurate splitting is the baseline for any serious RAG system.