LlamaIndex data connectors: a complete overview

This llamaindex data connectors overview tutorial walks through the ordered path from raw bytes to indexed nodes, using the loaders LlamaIndex actually ships. You’ll see how to read local files, hit external APIs, and ingest from databases without rewriting your pipeline for each source.

Before you start

Install the core package and the reader extras you need. LlamaIndex split loaders into optional dependencies to keep the base install small.

pip install llama-index-core llama-index-readers-file
pip install llama-index-readers-notion llama-index-readers-database

Pin versions in production. The reader APIs shift between minor releases, and a breaking change in NotionPageReader will silently drop documents if you don’t lock it. Use a constraints file or Poetry lock.

Step 1: Local files with SimpleDirectoryReader

Start here even if you think your data is “elsewhere.” Most teams have a dump of PDFs or Markdown that predates their SaaS stack. SimpleDirectoryReader handles extension detection and delegates to built-in file extractors.

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader(
    input_dir="./data",
    recursive=True,
    exclude=["*.tmp", "*.log"],
).load_data()

The call returns a list of Document objects. Each document carries text and a metadata dict with file_path and file_name. That metadata is your only hook for later filtering, so don’t discard it.

For non-standard extensions, pass a file_extractor map. A custom extractor is just a callable that takes a Path and returns a string.

from pathlib import Path

def epub_extractor(path: Path) -> str:
    # use ebooklib or similar
    return path.read_text(encoding="utf-8", errors="ignore")

reader = SimpleDirectoryReader(
    input_dir="./books",
    file_extractor={".epub": epub_extractor},
)

Pitfall: encoding errors on Windows-1252 files will raise and kill the whole batch. Set encoding="utf-8" and wrap the call in a try/except per file if you cannot trust the source. Also, recursive=True will descend into .git directories; exclude them explicitly.

Step 2: SaaS and API connectors via LlamaHub

Once local files are covered, pull live data from the tools your company actually uses. LlamaHub hosts dozens of readers; each is a small class with a load_data method. The second part of this llamaindex data connectors overview tutorial deals with structured sources, but first the API loaders.

Notion

Notion is the most common request. Create an internal integration, share pages with it, and pass the token.

from llama_index.readers.notion import NotionPageReader

reader = NotionPageReader(integration_token="secret_abc123")
docs = reader.load_data(page_ids=["1a2b3c", "4d5e6f"])

The reader fetches page blocks and concatenates them. It does not preserve nested database views well—expect to post-process if you rely on Notion databases rather than pages. Block types like toggles and synced blocks may render as empty strings.

Slack or Discord

For chat history, use the respective reader. They require bot tokens and rate-limit awareness.

from llama_index.readers.slack import SlackReader

reader = SlackReader(api_token="xoxb-...")
docs = reader.load_data(channel_ids=["C12345"], max_messages=2000)

Tradeoff: chat data is high-volume and low-signal. Apply a dedup pass before embedding, or you’ll burn tokens storing “thanks” and “lol”. Also, Slack’s free tier limits history depth; you may only see the last 90 days.

Step 3: Structured data from databases

Relational data often holds the highest-value context: customer records, support tickets, product catalogs. DatabaseReader runs a query and maps rows to documents.

from llama_index.readers.database import DatabaseReader

reader = DatabaseReader(
    scheme="postgresql",
    host="localhost",
    port="5432",
    user="readonly",
    password="pw",
    dbname="app",
)
docs = reader.load_data(query="SELECT title, body FROM articles WHERE published = true")

Each row becomes one document. Concatenate columns into a single text field with a clear separator, and push column names into metadata for filtering.

from llama_index.core import Document

rows = [(1, "Fix OOM", "Update pool size"), (2, "Latency", "Add cache")]
docs = [
    Document(
        text=f"Title: {t}\nBody: {b}",
        doc_id=f"ticket:{i}",
        metadata={"title": t},
    )
    for i, t, b in rows
]

Never use a privileged credential. The reader executes arbitrary SQL; a leaked token means a dropped table. Use a read replica.

Step 4: Scale and incremental ingestion

The load_data pattern works for thousands of documents. Past that, you need batching and checkpoints.

For file systems, iterate with os.walk and call SimpleDirectoryReader on subfolders of 500 files. For APIs, respect X-RateLimit-Remaining headers and sleep. For databases, page with WHERE id > :last_id LIMIT 1000.

LlamaIndex does not give you idempotent ingestion out of the box. Generate a stable doc_id from the source identifier (file path, page id, row primary key) and dedupe in your vector store.

from llama_index.core import Document

doc = Document(
    text=row["body"],
    doc_id=f"article:{row['id']}",
    metadata={"title": row["title"]},
)

Store the max processed ID in a small state table. On crash, restart from that offset. This pattern turns a fragile script into a resilient worker.

Step 5: Transformations, embeddings, and LLM calls

Raw documents are too large for most indexes. Split them and embed.

from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = splitter.get_nodes_from_documents(documents)
embed_model = OpenAIEmbedding()
for node in nodes:
    node.embedding = embed_model.get_text_embedding(node.text)

If your ingestion uses LLM-based summarization or an embedding model, route those calls through a single OpenAI-compatible endpoint like n4n.ai to get automatic fallback when a provider is rate-limited and per-token metering without code changes. This matters when a connector fires 10k embedding requests in a loop and OpenAI throws 429s.

Sentence splitting is heuristic. For code, use CodeSplitter with a language parser. For HTML, strip tags before splitting or you’ll chunk on markup.

Common pitfalls and tradeoffs

Metadata loss. Connectors default to minimal metadata. Add source, timestamp, and owner before indexing, or you’ll never answer “where did this come from?”

Chunk size guessing. 512 tokens is a starting point, not a rule. Code repositories need larger chunks; chat needs smaller. Measure retrieval hit rate on a eval set.

Rate limits. SaaS readers will hit API caps. Implement exponential backoff; the libraries don’t.

Cost. Embedding a 50MB PDF dump costs real money. Run a dry-run with len(text) counts before calling the model.

Duplicate content. Notion and Slack both return edited histories. Hash text and skip unchanged nodes.

Schema drift. Database columns rename. Pin a view in the DB instead of querying raw tables.

Minimal end-to-end pipeline

Here is a copy-paste skeleton that respects the above.

from llama_index.core import SimpleDirectoryReader, Document, SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding

def ingest(path: str):
    docs = SimpleDirectoryReader(path, recursive=True,
                                 exclude=["*.tmp", ".git/*"]).load_data()
    for d in docs:
        d.doc_id = f"file:{d.metadata['file_path']}"
    splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
    nodes = splitter.get_nodes_from_documents(docs)
    emb = OpenAIEmbedding()
    for n in nodes:
        n.embedding = emb.get_text_embedding(n.text)
    return nodes

if __name__ == "__main__":
    nodes = ingest("./data")
    print(f"Ingested {len(nodes)} nodes")

This llamaindex data connectors overview tutorial gave you the ordered path: local files, API loaders, databases, scaling, and transformation. Swap the first step for any LlamaHub reader and the rest of the pipeline stays identical.