LlamaIndex SimpleDirectoryReader for local file ingestion

This llamaindex simpledirectoryreader tutorial walks through ingesting local files into LlamaIndex using the SimpleDirectoryReader class. If you need to turn a folder of text, markdown, or PDFs into retrievable documents without writing a custom loader, this is the fastest path. We’ll build a small pipeline from scratch, inspect outputs at each step, and cover the configuration knobs that matter in practice.

Prerequisites

• Python 3.10 or newer.
• A virtual environment (recommended).
• llama-index installed (tested on 0.11.x). For PDF support also install pypdf.

python -m venv .venv
source .venv/bin/activate
pip install llama-index pypdf

No API keys are required for the loading steps. Embedding and indexing later will need an embedding endpoint; we’ll touch that at the end.

Scaffold a sample corpus

Create a data/ directory with mixed file types. Keep it small so output is readable.

mkdir -p data/subdir
printf "First line of plain text.\nSecond line with details.\n" > data/note.txt
printf "# Heading\nMarkdown body text for parsing.\n" > data/doc.md
printf "secret content\n" > data/secret.txt
printf "nested file in subdir\n" > data/subdir/extra.txt

You now have four files across two directories. This is enough to demonstrate recursion, extension filtering, and exclusion.

Load everything recursively

The default behavior of SimpleDirectoryReader is recursive and picks up .txt, .md, and several other formats via installed readers.

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
print(f"Loaded {len(documents)} documents")
for doc in documents:
    print(f"{doc.metadata['file_path']} -> {len(doc.text)} chars")

Expected output (paths may be absolute depending on OS):

Loaded 4 documents
data/note.txt -> 45 chars
data/doc.md -> 38 chars
data/secret.txt -> 15 chars
data/subdir/extra.txt -> 21 chars

Each item is a Document with .text and .metadata. The file_path key is populated automatically; file_name and creation_date are also added when the filesystem exposes them.

Filter by extension and exclude files

In any real corpus you’ll want to ignore temp files, .git, or secrets. Use required_exts and exclude.

reader = SimpleDirectoryReader(
    input_dir="data",
    required_exts=[".txt"],
    exclude=["data/secret.txt"],
    recursive=True,
)
filtered = reader.load_data()
print([d.metadata["file_name"] for d in filtered])

Output:

['note.txt', 'extra.txt']

The markdown file is dropped because it lacks the .txt extension, and secret.txt is explicitly excluded. This pattern is how you avoid ingesting junk during a quick llamaindex simpledirectoryreader tutorial experiment.

Load an explicit file list

Sometimes you already have a manifest. Pass input_files instead of input_dir:

reader = SimpleDirectoryReader(
    input_files=["data/note.txt", "data/doc.md"]
)
explicit = reader.load_data()
print(len(explicit), explicit[0].metadata["file_name"])

Output:

2 note.txt

When input_files is set, directory walking is skipped entirely.

Add custom metadata

Metadata drives filtering at retrieval time. Attach a source tag or owner via file_metadata.

def add_owner(filepath):
    return {"owner": "team-a" if "subdir" in filepath else "team-b"}

reader = SimpleDirectoryReader(
    input_dir="data",
    file_metadata=add_owner,
)
docs = reader.load_data()
for d in docs:
    print(d.metadata["file_name"], d.metadata["owner"])

Output:

note.txt team-b
doc.md team-b
secret.txt team-b
extra.txt team-a

The callable receives the absolute file path and returns a dict merged into each document’s metadata. Avoid heavy I/O in this function; it runs per file.

Control concurrency and limits

For large directories, use num_workers to parallelize reads and max_files to cap memory.

reader = SimpleDirectoryReader(
    input_dir="data",
    num_workers=4,
    max_files=2,
)
limited = reader.load_data()
print(len(limited))

Output is 2. The reader stops after yielding two files (order is not guaranteed). Use this when prototyping on a massive drop to avoid OOM.

Convert documents to nodes

Documents are whole files. For RAG you usually split them into smaller chunks called nodes. Use SentenceSplitter.

from llama_index.core.node_parser import SentenceSplitter

parser = SentenceSplitter(chunk_size=64, chunk_overlap=8)
nodes = parser.get_nodes_from_documents(documents)
print(f"Created {len(nodes)} nodes from {len(documents)} docs")
print(nodes[0].get_content()[:50])

With our tiny files this may produce one node per document, but on larger text you’ll see multiple chunks. chunk_size is approximate (token-based, not character-exact). Each node carries the parent document’s metadata plus a doc_id and node_id.

Wire into an embedding pipeline

Loading is only useful if you embed and index. LlamaIndex uses Settings.embed_model for this. If you want a single OpenAI-compatible endpoint for 240+ models with automatic fallback, you can point it at n4n.ai and keep your code unchanged.

from llama_index.core import Settings
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_base="https://api.n4n.ai/v1",
    api_key="YOUR_KEY",
)
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex(nodes)

The api_base swap is the only change needed; n4n.ai honors the same chat and embedding routes as OpenAI and forwards provider cache-control hints. That’s the whole integration surface.

Stream large corpora

load_data() reads everything into RAM. For giant trees, use iter_data() if your version supports it:

reader = SimpleDirectoryReader(input_dir="data", required_exts=[".txt"])
for doc in reader.iter_data():
    print(doc.metadata["file_name"], len(doc.text))

This yields documents one at a time, letting you persist or embed incrementally.

Common pitfalls

• Hidden files: SimpleDirectoryReader ignores files starting with . by default. If you need them, pre-collect paths and use input_files.
• Symlinks: Not followed unless the OS resolves them; test on your platform.
• PDF without pypdf: You’ll get a ValueError about missing dependencies. Install the reader extra.
• Metadata drift: creation_date may be empty on some filesystems; don’t assume it exists in downstream filters.
• Chunk size confusion: SentenceSplitter does not guarantee exact sizes; validate node lengths before embedding.

When not to use it

Don’t use SimpleDirectoryReader as a cloud connector or web crawler. It is a local filesystem reader. For S3, Notion, or Slack use the dedicated LlamaIndex readers. For permission-aware enterprise shares, wrap your own walker and feed paths to input_files.

Quick reference

from llama_index.core import SimpleDirectoryReader

# Minimal
docs = SimpleDirectoryReader("data").load_data()

# Production-leaning
docs = SimpleDirectoryReader(
    input_dir="data",
    required_exts=[".txt", ".md"],
    exclude=["data/.cache"],
    recursive=True,
    num_workers=8,
    max_files=5000,
    file_metadata=lambda p: {"path": p},
).load_data()

This llamaindex simpledirectoryreader tutorial covered the loader end-to-end: from a bare directory to filtered, metadata-rich documents and nodes ready for embedding. The class is deliberately simple—push it harder with custom readers only when you hit its limits.