This llamaindex ingestion pipeline tutorial walks through building a reproducible flow from raw files to queryable nodes. We’ll use LlamaIndex’s IngestionPipeline class, show where each stage breaks in production, and how to fix it without rewriting your RAG stack.
What the pipeline actually does
LlamaIndex ingests by running a sequence of transformations over Document objects. The pipeline replaces ad-hoc scripts that read files, chunk text, call an LLM, and embed in one messy function. It gives you a declarative list of steps and a consistent Node output that vector stores understand.
The core abstraction is simple:
from llama_index.core.ingestion import IngestionPipeline
pipeline = IngestionPipeline(transformations=[...])
nodes = pipeline.run(documents=docs)Each transformation is callable on a list of nodes. You can compose splitters, extractors, and embedding models. The pipeline does not hide magic; it orders your bugs.
Step 1: Load data with a connector
Start with SimpleDirectoryReader for local files. It auto-detects extensions and picks a reader.
from llama_index.core import SimpleDirectoryReader
docs = SimpleDirectoryReader(
input_dir="./data",
recursive=True,
exclude=["*.tmp", "*.DS_Store"],
).load_data()For anything beyond files, use a dedicated connector. LlamaIndex ships readers for Slack, Notion, PostgreSQL, and S3. Write a custom BaseReader if your source is internal.
Pitfall: SimpleDirectoryReader silently skips unknown extensions. If your .log files vanish, register a reader or rename. Always log len(docs) immediately after load.
Step 2: Parse into documents and nodes
A Document is raw loaded content plus metadata. The pipeline converts documents to Node objects via a node parser. The most common is SentenceSplitter.
from llama_index.core.node_parser import SentenceSplitter
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)Chunk size is measured in characters, not tokens. If you target a 4k token context, 512 chars is conservative for English (~100 tokens). Overlap prevents sentence fragmentation across boundaries.
Tradeoff: Small chunks improve retrieval precision but multiply embedding calls. Large chunks retain context but dilute similarity scores. Start at 512/64 and tune against real queries.
Step 3: Chunking with purpose
Do not use the default splitter blindly on code or tables. Code should split on function boundaries; CSV should stay row-grouped. Use CodeSplitter from llama_index.core.node_parser for repos.
from llama_index.core.node_parser import CodeSplitter
code_splitter = CodeSplitter(
language="python",
chunk_lines=40,
chunk_lines_overlap=10,
)For structured data, consider not chunking at all—store the full row as a node with metadata filters.
Step 4: Enrich with metadata extractors
Raw text chunks are weak signals. Extractors add title, keywords, or hypothetical questions. These run an LLM per node, so cost scales with chunk count.
from llama_index.core.extractors import TitleExtractor, QuestionsAnsweredExtractor
from llama_index.llms.openai import OpenAI
llm = OpenAI(model="gpt-4o-mini")
extractors = [
TitleExtractor(llm=llm, nodes=5),
QuestionsAnsweredExtractor(llm=llm, questions=3),
]If you need an LLM for extractors, point LlamaIndex at an OpenAI-compatible endpoint. Swapping in a gateway like n4n.ai gives you automatic fallback across providers when one is rate-limited, without changing extractor code.
Pitfall: Extractors add latency equal to num_nodes × extractor_calls. For 10k nodes, that’s a long batch. Run extractors only on the first pass, then cache nodes to disk with DocstoreStrategy.
Step 5: Embed and persist
Embeddings turn nodes into vectors. Use resolve_embed_model for local or hosted models.
from llama_index.core.embeddings import resolve_embed_model
embed_model = resolve_embed_model("local:BAAI/bge-small-en-v1.5")Wire everything together and persist to a vector index:
from llama_index.core.ingestion import IngestionPipeline, DocstoreStrategy
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.storage.index_store import SimpleIndexStore
from llama_index.core.vector_stores import SimpleVectorStore
pipeline = IngestionPipeline(
transformations=[splitter, *extractors, embed_model],
docstore=SimpleDocumentStore(strategy=DocstoreStrategy.UPSERT),
vector_store=SimpleVectorStore(),
index_store=SimpleIndexStore(),
)
nodes = pipeline.run(documents=docs)DocstoreStrategy.UPSERT deduplicates by node ID. Re-running the script updates changed files instead of duplicating vectors.
Common pitfalls and tradeoffs
Idempotency: Without a docstore, every run appends. Use UPSERT or hash-based IDs.
Metadata loss: Connectors attach file_path and file_name. If you drop metadata in custom parsers, you lose filterability. Keep node.metadata["source"] alive.
Embedding drift: Changing embedding models invalidates the store. Version your index or rebuild.
LLM coupling: Extractors block on LLM rate limits. Batch with asyncio or set pipeline.run(show_progress=True) to spot stalls.
Cost: Extractors + embeddings on 100k docs can cost dollars in LLM calls alone. Profile with a 100-doc sample first.
A minimal end-to-end example
from llama_index.core import SimpleDirectoryReader
from llama_index.core.ingestion import IngestionPipeline, DocstoreStrategy
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.extractors import TitleExtractor
from llama_index.core.embeddings import resolve_embed_model
from llama_index.llms.openai import OpenAI
docs = SimpleDirectoryReader("./data").load_data()
pipe = IngestionPipeline(
transformations=[
SentenceSplitter(chunk_size=512, chunk_overlap=64),
TitleExtractor(llm=OpenAI(model="gpt-4o-mini")),
resolve_embed_model("local:BAAI/bge-small-en-v1.5"),
],
docstore=SimpleDocumentStore(strategy=DocstoreStrategy.UPSERT),
)
nodes = pipe.run(documents=docs)
print(f"Produced {len(nodes)} nodes")This llamaindex ingestion pipeline tutorial snippet is enough to index a folder locally. Swap SimpleVectorStore for Pinecone or pgvector in production.
When not to use the pipeline
If data arrives as a stream (Kafka, webhooks), the batch run call is wrong. Use pipeline.transform on single documents and push nodes to the store incrementally.
If you need per-file custom logic (e.g., redacting PII before chunking), subclass BaseNodeParser and insert it as the first transformation. The pipeline stays declarative; your logic stays isolated.
Tuning for retrieval quality
After ingestion, measure hit rate with llama_index.eval. If chunks are too fine, increase chunk_size. If titles are garbage, drop TitleExtractor and rely on file_name metadata.
The pipeline is not a silver bullet. It is a disciplined way to compose the same operations you’d hack together otherwise. In the later parts of this llamaindex ingestion pipeline tutorial, focus on the transformation list as your single source of truth—when retrieval fails, print the node text and metadata before blaming the model.
Operational checklist
- Log document count at load.
- Pin chunk size to your embedding model’s trained context.
- Use UPSERT to avoid duplicate vectors.
- Cache extractor output for reprocessing.
- Test with 1% of data before full run.
Follow those and the pipeline will scale past prototype without surprises.