Semantic chunking in LangChain with embeddings

Fixed-size chunking breaks sentences mid-thought and scatters related concepts across boundaries. LangChain semantic chunking embeddings solve this by splitting documents at natural semantic boundaries instead of character counts. This guide walks through implementing and tuning semantic chunking for production RAG pipelines, from basic setup to evaluation metrics that matter.

Step 1: understand why fixed-size chunking fails

Character-based splitters (RecursiveCharacterTextSplitter, TokenTextSplitter) operate on syntax, not meaning. A 1000-character window might slice through a code example, separate a function signature from its docstring, or split a logical argument across two chunks. Retrieval then returns incomplete context, forcing the model to hallucinate connections.

Semantic chunking uses embedding similarity to detect topic shifts. Adjacent sentences with high cosine similarity stay together; a sharp drop signals a boundary. The result: chunks that map to coherent ideas, not arbitrary lengths.

Step 2: set up the environment

Install the minimal dependencies. We’ll use OpenAI embeddings for demonstration, but the pattern works with any provider.

pip install langchain langchain-openai langchain-community tiktoken numpy

Configure your API key. For production, use a secrets manager — never hardcode.

export OPENAI_API_KEY="sk-..."

Verify the imports work:

from langchain_openai import OpenAIEmbeddings
from langchain_experimental.text_splitter import SemanticChunker
from langchain_core.documents import Document
import numpy as np

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
print("Embeddings dimension:", len(embeddings.embed_query("test")))

You should see 1536 for text-embedding-3-small or 3072 for text-embedding-3-large.

Step 3: basic semantic chunking with SemanticChunker

LangChain’s SemanticChunker (in langchain_experimental) wraps the breakpoint detection logic. Start with defaults:

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
chunker = SemanticChunker(embeddings)

sample_text = """
LangChain provides a standard interface for chains, agents, and memory.
Developers can compose these primitives into complex applications.

The library supports multiple model providers including OpenAI, Anthropic, and local models.
Switching providers requires minimal code changes.

Vector stores enable semantic search over documents.
Popular options include Chroma, Pinecone, Weaviate, and FAISS.
Each has different tradeoffs for latency, cost, and scalability.
"""

chunks = chunker.create_documents([sample_text])
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1} ({len(chunk.page_content)} chars):")
    print(chunk.page_content[:120] + "...")
    print("---")

Output shows three chunks aligned with the paragraph topics. The splitter detected the shifts from “interface primitives” to “model providers” to “vector stores.”

How the default breakpoint detection works

SemanticChunker computes embeddings for each sentence, then calculates cosine similarity between adjacent sentences. The default breakpoint_threshold_type="percentile" with breakpoint_threshold_amount=95 marks a boundary when similarity drops below the 5th percentile of all pairwise similarities in the document.

This adaptive threshold handles documents with varying internal coherence better than a fixed cosine cutoff.

Step 4: tune breakpoint thresholds for your domain

Default percentiles work for general text but often over-split technical documentation or under-split narrative prose. Adjust based on your corpus.

Percentile threshold (recommended starting point)

chunker = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=85,  # lower = more chunks
)

Lower percentile → more aggressive splitting. For API reference docs where each method deserves its own chunk, try 70-80. For blog posts with flowing arguments, 90-95 preserves context.

Standard deviation threshold

chunker = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="standard_deviation",
    breakpoint_threshold_amount=1.5,  # boundaries at mean - 1.5*std
)

This marks boundaries where similarity drops more than N standard deviations below the mean. Useful when you want statistically significant topic shifts only.

Interquartile range threshold

chunker = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="interquartile",
    breakpoint_threshold_amount=1.5,  # boundaries below Q1 - 1.5*IQR
)

Robust to outliers. Preferred for noisy corpora (scraped web pages, OCR output) where a few anomalous sentences shouldn’t skew the threshold.

Gradient threshold (experimental)

chunker = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="gradient",
    breakpoint_threshold_amount=0.3,  # minimum gradient magnitude
)

Detects the steepest drops in similarity. Sensitive to local structure but can over-split on noise. Test thoroughly.

Step 5: use custom embedding models

OpenAI embeddings work well out of the box, but you may need local models for privacy, cost, or domain adaptation.

Local embeddings with Hugging Face

from langchain_community.embeddings import HuggingFaceEmbeddings

local_embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2",
    model_kwargs={"device": "cpu"},  # or "cuda"
    encode_kwargs={"normalize_embeddings": True},
)

chunker = SemanticChunker(local_embeddings, breakpoint_threshold_amount=85)

all-MiniLM-L6-v2 (384 dimensions) runs fast on CPU. For higher quality, try BAAI/bge-base-en-v1.5 (768 dims) or intfloat/e5-base-v2.

Domain-adapted embeddings

If you have labeled data, fine-tune an embedding model on your domain (legal, medical, code). The semantic chunker will then detect boundaries aligned with your domain’s conceptual structure.

# Example: using a fine-tuned model from your registry
domain_embeddings = HuggingFaceEmbeddings(
    model_name="your-org/legal-bert-embeddings",
    encode_kwargs={"normalize_embeddings": True},
)
chunker = SemanticChunker(domain_embeddings, breakpoint_threshold_amount=80)

Normalize embeddings (normalize_embeddings=True) so cosine similarity equals dot product — required for the percentile math to behave correctly.

Step 6: evaluate chunk quality quantitatively

Don’t guess. Measure.

Metric 1: chunk size distribution

def analyze_chunks(chunks):
    lengths = [len(c.page_content) for c in chunks]
    print(f"Count: {len(lengths)}")
    print(f"Mean: {np.mean(lengths):.0f} chars")
    print(f"Median: {np.median(lengths):.0f} chars")
    print(f"Std: {np.std(lengths):.0f}")
    print(f"Min/Max: {min(lengths)}/{max(lengths)}")
    print(f"90th percentile: {np.percentile(lengths, 90):.0f}")

analyze_chunks(chunks)

Target: median chunk size near your retrieval window (typically 500-1500 tokens for most LLMs). High variance suggests threshold needs tuning.

Metric 2: semantic coherence within chunks

from sklearn.metrics.pairwise import cosine_similarity

def chunk_coherence(chunks, embeddings):
    scores = []
    for chunk in chunks:
        sentences = chunk.page_content.split(". ")
        if len(sentences) < 2:
            continue
        sent_embeddings = embeddings.embed_documents(sentences)
        sims = cosine_similarity(sent_embeddings)
        # Mean pairwise similarity excluding diagonal
        mask = ~np.eye(len(sims), dtype=bool)
        scores.append(sims[mask].mean())
    return np.mean(scores)

print(f"Mean intra-chunk similarity: {chunk_coherence(chunks, embeddings):.3f}")

Higher is better. Compare across threshold settings. A drop below ~0.65 (for OpenAI embeddings) often indicates over-splitting.

Metric 3: boundary quality — does the split make sense?

def inspect_boundaries(chunks, n=3):
    for i in range(min(n, len(chunks) - 1)):
        end = chunks[i].page_content[-200:]
        start = chunks[i + 1].page_content[:200]
        print(f"--- Boundary {i+1} ---")
        print(f"Prev ends: ...{end}")
        print(f"Next starts: {start}...")
        print()

inspect_boundaries(chunks)

Manual review of 10-20 boundaries catches systematic errors (e.g., code blocks always split, headers separated from content).

Metric 4: retrieval precision (end-to-end test)

from langchain_community.vectorstores import FAISS
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA

# Build index
vectorstore = FAISS.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

# Test queries with known answers
test_cases = [
    ("What providers does LangChain support?", "OpenAI, Anthropic, and local models"),
    ("Which vector stores are mentioned?", "Chroma, Pinecone, Weaviate, and FAISS"),
    ("What is the standard interface for?", "chains, agents, and memory"),
]

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)

for query, expected in test_cases:
    result = qa.invoke({"query": query})
    print(f"Q: {query}")
    print(f"A: {result['result'][:200]}")
    print(f"Expected: {expected}")
    print("---")

Score each answer: does the retrieved context contain the answer? This is the only metric that ultimately matters.

Step 7: handle edge cases in production

Minimum and maximum chunk sizes

Semantic chunking can produce tiny chunks (single sentences) or massive ones (entire chapters). Clamp the range:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Post-process: merge too-small chunks, split too-large ones
def normalize_chunks(chunks, min_chars=200, max_chars=2000):
    normalized = []
    buffer = ""
    
    for chunk in chunks:
        buffer += chunk.page_content + "\n\n"
        
        while len(buffer) >= max_chars:
            # Split oversized buffer with recursive splitter
            splitter = RecursiveCharacterTextSplitter(
                chunk_size=max_chars,
                chunk_overlap=200,
            )
            parts = splitter.split_text(buffer)
            normalized.extend([Document(page_content=p) for p in parts[:-1]])
            buffer = parts[-1]
        
        if len(buffer) >= min_chars:
            normalized.append(Document(page_content=buffer.strip()))
            buffer = ""
    
    if buffer and len(buffer) >= min_chars:
        normalized.append(Document(page_content=buffer.strip()))
    
    return normalized

Preserve metadata through chunking

source_doc = Document(
    page_content=sample_text,
    metadata={"source": "langchain-docs.md", "section": "overview", "version": "0.1.0"}
)

chunks = chunker.create_documents([source_doc.page_content])
# Metadata is NOT automatically propagated — attach it:
for chunk in chunks:
    chunk.metadata = source_doc.metadata.copy()
    chunk.metadata["chunk_index"] = chunks.index(chunk)

Handle structured content (code, tables, lists)

Semantic chunkers struggle with code because embeddings treat code as prose. Pre-split structured blocks:

import re

def pre_split_structured(text):
    """Extract code blocks, tables, and lists before semantic chunking."""
    # Pattern for fenced code blocks
    code_pattern = r"(```[\s\S]*?```)"
    parts = re.split(code_pattern, text)
    
    result = []
    for part in parts:
        if part.startswith("```"):
            result.append(("code", part))
        elif part.strip():
            result.append(("text", part))
    return result

def chunk_mixed_content(text, chunker):
    segments = pre_split_structured(text)
    all_chunks = []
    
    for seg_type, content in segments:
        if seg_type == "code":
            # Keep code blocks intact or split by function/class
            all_chunks.append(Document(page_content=content, metadata={"type": "code"}))
        else:
            chunks = chunker.create_documents([content])
            for c in chunks:
                c.metadata["type"] = "text"
            all_chunks.extend(chunks)
    
    return all_chunks

Step 8: integrate into a LangChain RAG pipeline

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

# Build the full chain
prompt = ChatPromptTemplate.from_template("""
Answer the question using only the provided context.
If the context doesn't contain the answer, say you don't know.

Context:
{context}

Question: {question}
""")

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
)

# Test
response = rag_chain.invoke("How do I switch model providers in LangChain?")
print(response.content)

Step 9: monitor and iterate in production

Log chunking decisions for later analysis:

import json
import time

def chunk_with_logging(text, chunker, doc_id):
    start = time.time()
    chunks = chunker.create_documents([text])
    duration = time.time() - start
    
    log_entry = {
        "doc_id": doc_id,
        "timestamp": time.time(),
        "input_chars": len(text),
        "num_chunks": len(chunks),
        "chunk_sizes": [len(c.page_content) for c in chunks],
        "duration_ms": duration * 1000,
        "threshold_type": chunker.breakpoint_threshold_type,
        "threshold_amount": chunker.breakpoint_threshold_amount,
    }
    print(json.dumps(log_entry))  # Ship to your logging pipeline
    return chunks

Track these metrics over time:

• Chunks per document — sudden changes indicate upstream content shifts
• Mean chunk size — drift suggests embedding model or threshold issues
• Chunking latency — embedding calls dominate; batch if needed
• Retrieval precision — correlate with chunking parameters via A/B tests

Verification checklist

Before you deploy, confirm:

1. Chunk size distribution matches your retrieval window (median 500-1500 tokens, 90th percentile < 2000)
2. Intra-chunk similarity > 0.65 for your embedding model
3. Boundary inspection shows clean topic transitions, not mid-sentence or mid-code-block splits
4. End-to-end retrieval answers your test queries correctly with k=4
5. Latency is acceptable (semantic chunking adds 50-200ms per document vs. 1-5ms for character splitting)
6. Metadata propagates correctly through the pipeline
7. Structured content (code, tables) is handled by your pre-splitter

Common failure modes

Semantic chunking isn’t free — it adds embedding latency and complexity. But for RAG systems where retrieval quality determines answer quality, the improvement over fixed-size splitting is measurable and often decisive. Start with percentile 85, measure your coherence and retrieval metrics, and iterate from there.