Splitting code files in LangChain for RAG pipelines

When building RAG systems over codebases, naive text splitting destroys the semantic structure that makes code retrievable. LangChain’s code splitters preserve function boundaries, class definitions, and import statements by parsing syntax trees instead of counting characters. This guide walks through selecting, configuring, and validating a langchain code splitter rag pipeline that actually works in production.

Step 1: understand why character splitters fail on code

Character-based splitters like RecursiveCharacterTextSplitter treat code as plain text. They break mid-function, separate decorators from their targets, and scatter related logic across chunks. The result: retrieval returns fragments that don’t compile, lack context, and hallucinate when fed to an LLM.

Language-aware splitters use tree-sitter grammars to parse source files into abstract syntax trees (ASTs), then split along syntactic boundaries — functions, classes, methods, imports. This preserves the semantic units developers actually search for.

Step 2: install the required dependencies

LangChain’s code splitters live in langchain-text-splitters and require tree-sitter language packages. Install the base package plus the grammars for your target languages:

pip install langchain-text-splitters tree-sitter tree-sitter-python tree-sitter-javascript tree-sitter-typescript tree-sitter-go tree-sitter-rust tree-sitter-java

Each tree-sitter-* package provides a compiled grammar. If your language isn’t listed, check PyPI for tree-sitter-<language> or build from the tree-sitter grammar repository.

Step 3: choose the right splitter for your language

LangChain provides Language enum constants and a factory function. The most common splitters:

from langchain_text_splitters import Language, RecursiveCharacterTextSplitter
from langchain_text_splitters.code import CodeSplitter

# Option 1: Language-specific splitter (recommended)
python_splitter = CodeSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=2000,      # max characters per chunk
    chunk_overlap=200,    # overlap to preserve context across boundaries
)

# Option 2: Generic recursive splitter with code-aware separators
# Use when language isn't supported or you need fallback behavior
generic_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=2000,
    chunk_overlap=200,
)

Supported Language values include: PYTHON, JAVASCRIPT, TYPESCRIPT, GO, RUST, JAVA, CPP, CSHARP, RUBY, PHP, SWIFT, KOTLIN, SCALA, SQL, HTML, CSS, MARKDOWN, LATEX, SOL, PROTO, DOCKERFILE.

The CodeSplitter (Option 1) uses tree-sitter directly and produces cleaner boundaries. The RecursiveCharacterTextSplitter.from_language (Option 2) uses language-aware separator lists as a fallback — faster but less precise.

Step 4: load code files with appropriate document loaders

Pair the splitter with a loader that preserves file paths and metadata. DirectoryLoader with TextLoader works for simple cases; UnstructuredFileLoader handles more formats but adds overhead.

from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_core.documents import Document
from pathlib import Path

def load_codebase(root: str, glob: str = "**/*.py") -> list[Document]:
    loader = DirectoryLoader(
        root,
        glob=glob,
        loader_cls=TextLoader,
        loader_kwargs={"encoding": "utf-8"},
        show_progress=True,
        use_multithreading=True,
    )
    docs = loader.load()
    
    # Enrich metadata with relative path and language hint
    for doc in docs:
        path = Path(doc.metadata["source"])
        doc.metadata["relative_path"] = str(path.relative_to(root))
        doc.metadata["language"] = path.suffix.lstrip(".")
    
    return docs

# Usage
documents = load_codebase("/path/to/repo", "**/*.py")
print(f"Loaded {len(documents)} files")

For polyglot repos, call load_codebase per language with the appropriate glob and language suffix, then combine the document lists.

Step 5: split documents while preserving metadata

Pass loaded documents through the splitter. Each output chunk inherits the source document’s metadata — critical for citation and filtering at query time.

from langchain_text_splitters.code import CodeSplitter
from langchain_text_splitters import Language

def split_code_documents(
    documents: list[Document],
    language: Language = Language.PYTHON,
    chunk_size: int = 2000,
    chunk_overlap: int = 200,
) -> list[Document]:
    splitter = CodeSplitter.from_language(
        language=language,
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
    )
    
    chunks = splitter.split_documents(documents)
    
    # Add chunk index for ordering and deduplication
    for i, chunk in enumerate(chunks):
        chunk.metadata["chunk_index"] = i
        chunk.metadata["chunk_size"] = len(chunk.page_content)
    
    return chunks

# Usage for Python files
python_chunks = split_code_documents(documents, Language.PYTHON)
print(f"Produced {len(python_chunks)} chunks")

Tuning chunk_size: Code chunks should fit within your embedding model’s context window and leave room for the query + retrieved chunks in the generation prompt. For text-embedding-3-large (8192 tokens) and a 4k generation context, 1500-2500 characters per chunk is a safe range. Overlap of 10-15% (200-300 chars) preserves cross-boundary context without excessive duplication.

Step 6: handle multi-language repositories

Real codebases mix languages. Build a dispatcher that routes each file to its language-specific splitter:

from langchain_text_splitters import Language
from typing import Dict, List

LANGUAGE_MAP: Dict[str, Language] = {
    ".py": Language.PYTHON,
    ".js": Language.JAVASCRIPT,
    ".ts": Language.TYPESCRIPT,
    ".tsx": Language.TYPESCRIPT,
    ".jsx": Language.JAVASCRIPT,
    ".go": Language.GO,
    ".rs": Language.RUST,
    ".java": Language.JAVA,
    ".cpp": Language.CPP,
    ".cs": Language.CSHARP,
    ".rb": Language.RUBY,
    ".php": Language.PHP,
    ".swift": Language.SWIFT,
    ".kt": Language.KOTLIN,
    ".scala": Language.SCALA,
    ".sql": Language.SQL,
}

def split_polyglot_documents(documents: list[Document]) -> list[Document]:
    all_chunks = []
    
    for doc in documents:
        ext = Path(doc.metadata["source"]).suffix
        language = LANGUAGE_MAP.get(ext)
        
        if language is None:
            # Fallback: generic text splitter for unknown extensions
            from langchain_text_splitters import RecursiveCharacterTextSplitter
            fallback = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
            chunks = fallback.split_documents([doc])
        else:
            splitter = CodeSplitter.from_language(language=language, chunk_size=2000, chunk_overlap=200)
            chunks = splitter.split_documents([doc])
        
        for i, chunk in enumerate(chunks):
            chunk.metadata["chunk_index"] = i
            chunk.metadata["splitter_language"] = language.value if language else "generic"
        
        all_chunks.extend(chunks)
    
    return all_chunks

This approach ensures each language gets its proper AST parser while gracefully degrading for unsupported file types.

Step 7: embed and index chunks in a vector store

With chunks produced, embed and store them. This example uses FAISS for local development; swap the vector store for production (Pinecone, Weaviate, Qdrant, etc.).

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_core.vectorstores import VectorStore

def build_vector_store(chunks: list[Document], persist_path: str = "./code_index") -> VectorStore:
    embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
    
    # Batch embedding for throughput
    vector_store = FAISS.from_documents(chunks, embeddings)
    vector_store.save_local(persist_path)
    
    return vector_store

# Usage
vector_store = build_vector_store(python_chunks)
print(f"Indexed {vector_store.index.ntotal} vectors")

Metadata filtering: Include relative_path, language, and chunk_index in the vector store metadata. At query time, filter by language or path prefix to scope retrieval (e.g., only search src/auth/).

Step 8: verify split quality with automated checks

Don’t eyeball chunks. Write assertions that catch regressions when you change splitter config or upgrade tree-sitter grammars.

def verify_chunks(chunks: list[Document], sample_size: int = 50) -> dict:
    import random
    import ast
    
    results = {
        "total_chunks": len(chunks),
        "empty_chunks": 0,
        "oversized_chunks": 0,
        "syntax_errors": 0,
        "missing_metadata": 0,
        "boundary_issues": 0,
    }
    
    sample = random.sample(chunks, min(sample_size, len(chunks)))
    
    for chunk in sample:
        content = chunk.page_content
        
        # Check 1: no empty chunks
        if not content.strip():
            results["empty_chunks"] += 1
            continue
        
        # Check 2: size bounds
        if len(content) > 3000:  # adjust for your chunk_size + overlap
            results["oversized_chunks"] += 1
        
        # Check 3: required metadata
        required = ["source", "relative_path", "language", "chunk_index"]
        if not all(k in chunk.metadata for k in required):
            results["missing_metadata"] += 1
        
        # Check 4: valid Python syntax (for Python chunks)
        if chunk.metadata.get("language") == "py":
            try:
                ast.parse(content)
            except SyntaxError:
                results["syntax_errors"] += 1
        
        # Check 5: chunk doesn't start/end mid-token (heuristic)
        lines = content.strip().splitlines()
        if lines:
            first = lines[0].strip()
            last = lines[-1].strip()
            # Heuristic: chunks shouldn't start with 'else:', 'elif', 'except:', 'finally:'
            if first.startswith(("else:", "elif ", "except:", "finally:", "catch ", "}")):
                results["boundary_issues"] += 1
    
    return results

# Run verification
verification = verify_chunks(python_chunks)
print(verification)
# Target: all zeros except total_chunks

Add this to your CI pipeline. Fail the build if syntax_errors > 0 or boundary_issues > threshold.

Step 9: build a retrieval chain with metadata-aware search

A minimal RAG chain that filters by language and path:

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableLambda
from langchain_core.output_parsers import StrOutputParser
from operator import itemgetter

def build_rag_chain(vector_store: VectorStore, k: int = 6):
    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
    
    prompt = ChatPromptTemplate.from_messages([
        ("system", """You are a code assistant. Answer questions using only the provided context.
Cite sources using the relative_path and chunk_index from metadata.
If the context doesn't contain the answer, say you don't know."""),
        ("human", """Context:
{context}

Question: {question}

Answer:"""),
    ])
    
    def format_docs(docs: list[Document]) -> str:
        formatted = []
        for doc in docs:
            meta = doc.metadata
            header = f"[File: {meta.get('relative_path', 'unknown')}, Chunk: {meta.get('chunk_index', '?')}]"
            formatted.append(f"{header}\n```{meta.get('language', '')}\n{doc.page_content}\n```")
        return "\n\n---\n\n".join(formatted)
    
    retriever = vector_store.as_retriever(
        search_kwargs={"k": k}
        # Add filter: {"language": "py"} or {"relative_path": {"$regex": "^src/auth/"}}
    )
    
    chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )
    
    return chain

# Usage
rag_chain = build_rag_chain(vector_store)
answer = rag_chain.invoke("How does the authentication middleware validate JWT tokens?")
print(answer)

Step 10: evaluate retrieval quality with a golden set

Splitting quality ultimately shows up in retrieval metrics. Build a small golden set of (question, relevant_file_paths) pairs and measure recall@k.

from typing import List, Set

GOLDEN_SET = [
    {
        "question": "How does the rate limiter handle burst traffic?",
        "relevant_paths": {"src/rate_limiter/token_bucket.py", "src/rate_limiter/__init__.py"},
    },
    {
        "question": "Where are database migrations defined?",
        "relevant_paths": {"migrations/", "alembic/"},
    },
    {
        "question": "What configuration options does the auth module accept?",
        "relevant_paths": {"src/auth/config.py", "src/auth/__init__.py"},
    },
]

def evaluate_recall(vector_store: VectorStore, k: int = 6) -> dict:
    retriever = vector_store.as_retriever(search_kwargs={"k": k})
    
    total_recall = 0.0
    results = []
    
    for item in GOLDEN_SET:
        docs = retriever.invoke(item["question"])
        retrieved_paths = {doc.metadata.get("relative_path", "") for doc in docs}
        relevant = item["relevant_paths"]
        
        # Check if any relevant path is a prefix of retrieved path
        found = any(
            any(ret.startswith(rel) for ret in retrieved_paths)
            for rel in relevant
        )
        
        recall = 1.0 if found else 0.0
        total_recall += recall
        results.append({
            "question": item["question"],
            "recall": recall,
            "retrieved": list(retrieved_paths)[:3],
        })
    
    return {
        "mean_recall_at_k": total_recall / len(GOLDEN_SET),
        "details": results,
    }

# Run evaluation
eval_results = evaluate_recall(vector_store, k=6)
print(f"Recall@6: {eval_results['mean_recall_at_k']:.2f}")
for r in eval_results["details"]:
    print(f"  {r['recall']:.0f} - {r['question'][:60]}...")

Target recall@6 > 0.8 for a well-tuned splitter. If lower, adjust chunk_size, chunk_overlap, or add more golden set k.

Step 11: handle edge cases in production

Large generated files

Auto-generated code (protobufs, GraphQL schemas, bundled assets) can exceed chunk limits. Filter them at load time:

def should_skip_file(path: Path) -> bool:
    skip_patterns = [
        "**/node_modules/**",
        "**/dist/**",
        "**/build/**",
        "**/*.min.js",
        "**/*.pb.go",
        "**/*_pb2.py",
        "**/generated/**",
    ]
    # Use pathlib match or fnmatch
    return any(path.match(p) for p in skip_patterns)

Mixed-language files

Files like .vue, .svelte, or Jupyter notebooks contain multiple languages. The tree-sitter grammar for the primary language (HTML for Vue) will parse the whole file but may not split embedded script blocks cleanly. Options:

• Pre-process with a custom splitter that extracts <script> blocks
• Accept coarser chunks for these files
• Use UnstructuredFileLoader with mode="elements" for better segmentation

Encoding issues

TextLoader defaults to UTF-8. Legacy codebases may use Latin-1 or CP1252. Detect and handle:

import chardet

def load_with_detection(path: Path) -> Document:
    raw = path.read_bytes()
    detected = chardet.detect(raw)
    encoding = detected["encoding"] or "utf-8"
    content = raw.decode(encoding, errors="replace")
    return Document(page_content=content, metadata={"source": str(path)})

Step 12: monitor chunk distribution in production

Log chunk size distribution at index time. Skewed distributions indicate splitter misconfiguration.

import statistics

def log_chunk_stats(chunks: list[Document]) -> None:
    sizes = [len(c.page_content) for c in chunks]
    print(f"""
Chunk statistics:
  Count: {len(sizes)}
  Mean: {statistics.mean(sizes):.0f}
  Median: {statistics.median(sizes):.0f}
  Stdev: {statistics.stdev(sizes):.0f}
  Min: {min(sizes)}
  Max: {max(sizes)}
  P95: {statistics.quantiles(sizes, n=20)[18]:.0f}
  P99: {statistics.quantiles(sizes, n=100)[98]:.0f}
""")

log_chunk_stats(python_chunks)

Watch for bimodal distributions (two distinct chunk sizes) — usually means some files use a fallback splitter.

Common pitfalls and fixes

Final checklist before deploying

• All target languages have tree-sitter-* packages installed
• verify_chunks passes with zero syntax errors
• Golden set recall@6 > 0.8
• Chunk size distribution shows single mode around target size
• Metadata includes relative_path, language, chunk_index on every chunk
• Retriever supports metadata filtering for scoped queries
• CI runs verification on every index rebuild

The langchain code splitter rag pattern works because it respects the semantic boundaries that developers actually reason about. Treat the splitter as a tuned component — version your tree-sitter grammars, monitor chunk quality metrics, and evaluate retrieval like any other ML pipeline stage.