Fixed-size chunking vs semantic chunking for RAG

Fixed-size chunking splits documents into equal token windows with a sliding overlap. Semantic chunking uses an embedding model or heuristic to cut at topic boundaries. The choice determines whether your retrieval surface is predictable or meaningful — and it cascades into every downstream metric from latency to hallucination rate.

What fixed-size chunking does

Fixed-size chunking is deterministic. You pick a token budget (say 512 tokens) and an overlap (say 50 tokens), then slice sequentially. No external dependencies, no model calls, no nondeterminism.

def fixed_chunks(text: str, tokenizer, chunk_size: int = 512, overlap: int = 50) -> list[str]:
    tokens = tokenizer.encode(text)
    chunks = []
    for i in range(0, len(tokens), chunk_size - overlap):
        chunk_tokens = tokens[i:i + chunk_size]
        chunks.append(tokenizer.decode(chunk_tokens))
    return chunks

The overlap mitigates boundary cuts — a sentence split across chunks appears in both. But the strategy is blind to document structure. A code block, a table, and a paragraph all get the same treatment. If your chunk size is smaller than a typical semantic unit, you fragment context. If it’s larger, you dilute signal with noise.

What semantic chunking does

Semantic chunking attempts to cut where meaning shifts. The most common production approach: embed sentences (or paragraphs), compute cosine similarity between adjacent windows, and split where similarity drops below a threshold.

def semantic_chunks(
    text: str,
    embedder,
    threshold: float = 0.75,
    min_chunk_tokens: int = 100,
    max_chunk_tokens: int = 1000
) -> list[str]:
    sentences = split_sentences(text)  # your sentence splitter
    embeddings = embedder.embed(sentences)
    
    chunks = []
    current = [sentences[0]]
    current_tokens = count_tokens(sentences[0])
    
    for i in range(1, len(sentences)):
        sim = cosine(embeddings[i-1], embeddings[i])
        next_tokens = count_tokens(sentences[i])
        
        if sim < threshold and current_tokens >= min_chunk_tokens:
            chunks.append(" ".join(current))
            current = [sentences[i]]
            current_tokens = next_tokens
        elif current_tokens + next_tokens > max_chunk_tokens:
            chunks.append(" ".join(current))
            current = [sentences[i]]
            current_tokens = next_tokens
        else:
            current.append(sentences[i])
            current_tokens += next_tokens
    
    if current:
        chunks.append(" ".join(current))
    return chunks

Variants exist: some use hierarchical clustering, others use an LLM to propose boundaries, others rely on document markup (headings, HTML tags). The common thread: chunk boundaries correlate with semantic boundaries.

Comparison across dimensions

Retrieval quality

Fixed-size chunking produces uniform vectors. Every chunk has the same dimensional footprint, which makes ANN index behavior predictable. But a query about “authentication flow” might match the middle of a chunk that starts with “logging configuration” and ends with “rate limiting” — the relevant signal is diluted.

Semantic chunking aligns chunks with concepts. A query about authentication retrieves the authentication chunk, not a fragment. In practice this lifts recall@k for concept-oriented queries by 10-20% on benchmarks like HotpotQA and NarrativeQA. The trade-off: chunk lengths vary, so vector norms vary, which can bias similarity scores unless you normalize.

Latency and throughput

Fixed-size chunking is effectively free at ingestion. Tokenization is fast; slicing is O(n). You can chunk 10,000 documents per second on a single CPU core.

Semantic chunking requires embedding every sentence (or window). A typical BGE-small or MiniLM call runs ~2-5 ms per sentence on CPU, ~0.5 ms on GPU. For a 50-page PDF (~2,000 sentences), that’s 4-10 seconds of pure embedding time. Batch inference helps, but ingestion latency becomes a real pipeline concern. If you’re indexing millions of documents, you need a GPU fleet or you accept hours of lag.

Cost model

Fixed-size: near-zero compute cost. Storage cost is predictable — chunk count = total_tokens / (chunk_size - overlap).

Semantic: embedding compute dominates. At current prices, embedding 1M tokens with a small model costs ~$0.01-0.02 on managed APIs, or GPU-hours self-hosted. For a 100GB corpus, that’s hundreds of dollars per re-index. Variable chunk counts also make storage forecasting harder.

Ergonomics and debugging

Fixed-size is trivial to debug. Chunk 47 is tokens 23,500-24,000. You can grep the source, count tokens, verify overlap. When retrieval fails, you know exactly what the model saw.

Semantic chunking is opaque. A chunk boundary might fall in the middle of a code example because the embedder thought “def authenticate” and “return token” were dissimilar. You need tooling to visualize boundaries, inspect similarity scores, and trace why a query matched (or didn’t). Most teams underinvest here and pay in debugging time.

Ecosystem and tooling

LangChain, LlamaIndex, and Haystack all ship fixed-size splitters as defaults. They’re battle-tested, configurable, and work with any tokenizer.

Semantic splitters exist in the same libraries but are younger. LangChain’s SemanticChunker uses embedding similarity; LlamaIndex has SemanticSplitterNodeParser. Both require you to provide an embedder and tune thresholds. Community knowledge is thinner — fewer Stack Overflow answers, fewer GitHub issues to search.

Limits and failure modes

Fixed-size fails on:

Structured data (tables, code, JSON) where a split breaks parseability
Long coherent passages where overlap isn’t enough to preserve context
Queries needing multi-hop reasoning across artificial boundaries

Semantic chunking fails on:

Short documents where there aren’t enough sentences to detect boundaries
Domain-specific language where general-purpose embedders misjudge similarity
Adversarial formatting (bullet lists, FAQs) where every sentence looks dissimilar
Drift: if you switch embedders, all boundaries change — re-indexing is mandatory

Comparison table

Dimension	Fixed-size chunking	Semantic chunking
Ingestion latency	~ms per doc	~seconds per doc (embedding-bound)
Compute cost	Negligible	Embedding API calls or GPU-hours
Retrieval recall (concept queries)	Baseline	+10-20% typical
Chunk length variance	Zero	High (depends on threshold)
Debuggability	Trivial — deterministic offsets	Requires visualization tooling
Re-indexing on embedder change	Not needed	Mandatory
Structured data handling	Poor — breaks tables/code	Better if embedder respects structure
Default library support	Mature, all frameworks	Available, less battle-tested
Storage predictability	Exact formula	Variable, depends on corpus

Which to choose

Choose fixed-size when

You’re prototyping or iterating fast. Fixed-size gets you a working RAG pipeline in minutes. Swap to semantic later if retrieval quality demands it.

Your corpus is code, logs, or structured data. Semantic chunkers trained on natural language butcher source code. Fixed-size with a code-aware tokenizer (tree-sitter, Pygments) preserves function boundaries better than a general embedder.

Ingestion throughput matters more than recall. High-volume pipelines (millions of docs/day) can’t absorb embedding latency per document. Fixed-size scales horizontally with zero GPU budget.

You need reproducible, auditable chunking. Compliance, legal, or forensic workflows often require deterministic mapping from chunk to source offset.

Choose semantic when

Queries are concept-oriented, not keyword-oriented. Users ask “how does authentication work?” not “authentication function.” Semantic chunks match the mental model.

Documents are long-form prose: manuals, research papers, legal contracts. These have clear topic shifts. Fixed-size chops arguments mid-paragraph; semantic chunking keeps sections intact.

You have GPU budget and can tolerate ingestion latency. If you’re indexing nightly, the embedding cost is amortized. The recall gain compounds across every query.

You’re building a customer-facing product where retrieval quality is a differentiator. The 15% recall lift translates to fewer “I couldn’t find that” support tickets.

Hybrid approach (what most production systems converge to)

Use document structure first, then fixed-size within sections.

def hybrid_chunks(doc: Document, tokenizer, chunk_size: int = 512, overlap: int = 50) -> list[str]:
    # Split by headings, code fences, tables — whatever your parser exposes
    sections = doc.split_by_structure()
    chunks = []
    for section in sections:
        if section.is_structured:  # code, table, list
            chunks.append(section.text)  # keep atomic
        else:
            chunks.extend(fixed_chunks(section.text, tokenizer, chunk_size, overlap))
    return chunks

This preserves semantic boundaries where markup exists (headings, <table>, python fences) and falls back to fixed-size for plain prose. No embedding calls at ingestion. You get 80% of the semantic benefit for 5% of the cost.

If you later need finer granularity inside prose sections, run semantic chunking as a second pass on just the unstructured text — but only after you’ve measured that fixed-size within sections is actually hurting recall.

The honest answer: start with fixed-size. Measure recall on your actual query log. If concept queries miss, add structure-aware splitting. Only reach for embedding-based semantic chunking when you have evidence that structure-aware fixed-size isn’t enough, and you have the GPU budget to pay for it.