LlamaIndex web page ingestion with SimpleWebPageReader

If you’re building a RAG system that needs to pull content from live websites, the llamaindex simplewebpagereader tutorial you’re reading right now will get you from zero to a queryable index in about thirty minutes. SimpleWebPageReader is LlamaIndex’s built-in connector for fetching and parsing HTML into Document objects, and it handles the messy parts — redirects, encoding, script stripping — without pulling in heavy scraping frameworks. The following steps assume Python 3.10+ and a virtual environment you control.

Step 1: Install dependencies

Start with the core LlamaIndex package and the reader extras. If you plan to use OpenAI embeddings (the default in most examples), include that integration too.

pip install llama-index llama-index-readers-web llama-index-llms-openai llama-index-embeddings-openai

Verify the import works before moving on:

from llama_index.readers.web import SimpleWebPageReader
print(SimpleWebPageReader.__module__)
# llama_index.readers.web.simple_web

If that prints cleanly, your environment is ready.

Step 2: Basic ingestion with SimpleWebPageReader

The reader accepts a list of URLs and returns a list of Document objects. Each document contains the extracted text in text and metadata like the source URL in metadata.

from llama_index.readers.web import SimpleWebPageReader

urls = ["https://www.paulgraham.com/vb.html"]
reader = SimpleWebPageReader(html_to_text=True)
documents = reader.load_data(urls)

print(f"Loaded {len(documents)} document(s)")
print(f"First 500 chars:\n{documents[0].text[:500]}")
print(f"Metadata: {documents[0].metadata}")

Run this and you should see the essay text without HTML tags, plus a metadata dict containing at least url and title. The html_to_text=True flag (default) uses BeautifulSoup under the hood to strip scripts, styles, and navigation chrome.

Step 3: Handle multiple URLs and metadata

Real workloads involve batches of URLs. Pass the full list in one call — the reader processes them sequentially and preserves order.

urls = [
    "https://www.paulgraham.com/vb.html",
    "https://www.paulgraham.com/avg.html",
    "https://www.paulgraham.com/growth.html",
]

documents = reader.load_data(urls)

for i, doc in enumerate(documents):
    print(f"Doc {i}: {doc.metadata.get('title', 'no title')} — {len(doc.text)} chars")

Each document carries the original URL and the page <title> tag. If you need custom metadata (for example, a category or crawl timestamp), attach it after loading:

from datetime import datetime

for doc in documents:
    doc.metadata["crawled_at"] = datetime.utcnow().isoformat()
    doc.metadata["source_type"] = "blog"

This pattern keeps provenance intact for downstream filtering.

Step 4: Configure HTML parsing and cleaning

SimpleWebPageReader exposes a few knobs for sites that need special handling. The most useful are html_to_text, metadata_fn, and passing a custom requests session for auth or headers.

Custom metadata extraction

Use metadata_fn to pull arbitrary data from the parsed BeautifulSoup object. This example grabs the author from a meta tag and the publication date from JSON-LD:

from bs4 import BeautifulSoup
import json

def extract_extra_metadata(soup: BeautifulSoup, url: str) -> dict:
    meta = {}
    # Author from meta tag
    author_tag = soup.find("meta", attrs={"name": "author"})
    if author_tag and author_tag.get("content"):
        meta["author"] = author_tag["content"]
    # JSON-LD structured data
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string)
            if isinstance(data, dict) and data.get("@type") == "BlogPosting":
                meta["published_date"] = data.get("datePublished")
                break
        except Exception:
            pass
    return meta

reader = SimpleWebPageReader(html_to_text=True, metadata_fn=extract_extra_metadata)
documents = reader.load_data(urls)

for doc in documents:
    print(doc.metadata)

Custom request headers and sessions

Some sites block the default requests user agent or require cookies. Pass a pre-configured requests.Session:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (compatible; MyBot/1.0; +https://example.com/bot)"
})
# Add auth, cookies, etc. here if needed

reader = SimpleWebPageReader(html_to_text=True, requests_session=session)
documents = reader.load_data(["https://example.com/protected-page"])

If you hit rate limits or need JavaScript rendering, SimpleWebPageReader won’t help — you’ll need a headless browser approach (Playwright, Selenium) or a scraping API. That’s outside this connector’s scope.

Step 5: Chunk and index for retrieval

Raw documents are too large for direct embedding. Split them into nodes, embed, and persist a vector index. LlamaIndex’s SentenceSplitter works well for prose; adjust chunk_size and chunk_overlap for your content type.

from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

# Configure global settings (or pass explicitly to constructors)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)

# Build index from documents
index = VectorStoreIndex.from_documents(documents, show_progress=True)

# Persist to disk for reuse
index.storage_context.persist(persist_dir="./storage/web_index")

The show_progress=True flag gives you a tqdm bar — handy when ingesting hundreds of pages.

Alternative: Ingest directly into a managed vector store

If you’re using Pinecone, Weaviate, Qdrant, or another managed store, swap the storage context:

from llama_index.vector_stores.pinecone import PineconeVectorStore
from pinecone import Pinecone

pc = Pinecone(api_key="your-key")
pinecone_index = pc.Index("llamaindex-web")
vector_store = PineconeVectorStore(pinecone_index=pinecone_index)

from llama_index.core import StorageContext

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    show_progress=True
)

No local persist_dir needed; the index lives in your vector database.

Step 6: Verify the pipeline works

Don’t assume ingestion succeeded. Run a quick retrieval smoke test before declaring victory.

# Load persisted index (if using local storage)
from llama_index.core import load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="./storage/web_index")
index = load_index_from_storage(storage_context)

# Query
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("What does Paul Graham say about venture capital?")

print(response.response)
print("\n--- Sources ---")
for node in response.source_nodes:
    print(f"  Score: {node.score:.3f} | URL: {node.metadata.get('url')} | Title: {node.metadata.get('title')}")

You should see a coherent answer grounded in the ingested essays, with source nodes pointing back to the original URLs. If the response is generic or hallucinated, check:

1. Chunk size — too small loses context; too large dilutes relevance. Start at 512/50 and tune.
2. Embedding model — text-embedding-3-small is fast and cheap; text-embedding-3-large improves recall on technical content.
3. Top-k — increase to 5-10 for broader coverage, then rerank if needed.

Common pitfalls and fixes

Empty or truncated documents

If doc.text is empty or suspiciously short, the page likely requires JavaScript. SimpleWebPageReader only sees the initial HTML. Confirm with curl:

curl -sL "https://example.com/page" | head -c 2000

If the content you want isn’t in that output, you need a JS renderer. Option: use llama-index-readers-web’s PlaywrightWebPageReader (separate install) or fetch via a scraping API and feed raw HTML to SimpleWebPageReader.load_data with pre-fetched content.

Encoding issues

Non-UTF-8 pages sometimes slip through. Force encoding detection:

import requests

def fetch_with_encoding(url: str) -> str:
    resp = requests.get(url, timeout=30)
    resp.encoding = resp.apparent_encoding  # chardet guess
    return resp.text

html = fetch_with_encoding("https://example.com/weird-encoding")
# Then parse manually with BeautifulSoup and create Document objects directly

Duplicate content across URLs

Blog pagination, print views, and AMP pages create near-duplicates. Deduplicate before indexing:

from llama_index.core import Document

def deduplicate(documents: list[Document], threshold: float = 0.95) -> list[Document]:
    """Simple dedup by text hash prefix. For production, use MinHash or embedding similarity."""
    seen = set()
    unique = []
    for doc in documents:
        # Use first 200 chars as fingerprint
        fp = doc.text[:200].strip()
        if fp not in seen:
            seen.add(fp)
            unique.append(doc)
    return unique

documents = deduplicate(documents)

Rate limiting and politeness

SimpleWebPageReader has no built-in delay. Add one if you’re hitting many pages on the same domain:

import time
from urllib.parse import urlparse

def polite_load(reader: SimpleWebPageReader, urls: list[str], delay: float = 1.0) -> list[Document]:
    all_docs = []
    last_domain = None
    for url in urls:
        domain = urlparse(url).netloc
        if domain == last_domain:
            time.sleep(delay)
        docs = reader.load_data([url])
        all_docs.extend(docs)
        last_domain = domain
    return all_docs

documents = polite_load(reader, urls, delay=2.0)

Respect robots.txt and Crawl-Delay headers in production.

Scaling considerations

For a few hundred pages, the local flow above is fine. Beyond that, consider:

• Parallel fetching — Use asyncio with aiohttp and feed HTML to BeautifulSoup yourself, then construct Document objects. SimpleWebPageReader is synchronous.
• Incremental updates — Store URL → content hash (SHA256 of cleaned text) in a DB. On re-crawl, skip unchanged pages.
• Structured extraction — If you need specific fields (price, date, author) consistently, pair the reader with an LLM-based extractor (llama-index-extractors or custom Pydantic program) rather than relying on metadata_fn alone.

What this connector does not do

SimpleWebPageReader is intentionally narrow. It does not:

• Render JavaScript (no SPA support)
• Handle pagination or crawl discovery
• Manage cookies across multi-step flows
• Respect robots.txt automatically
• Provide retry/backoff policies

Treat it as a “fetch one URL, give me clean text” primitive. Build the crawler, scheduler, and politeness layer yourself — or use a dedicated scraping platform.

You now have a working ingestion pipeline: fetch → clean → chunk → embed → query. The same pattern applies whether you’re indexing documentation, news, competitor blogs, or internal wikis. Swap the embedding model, vector store, or chunking strategy as your latency and recall requirements evolve. The connector stays the same.