Multimodal RAG in LlamaIndex with GPT-4o and images

Retrieval-augmented generation stops being text-only when your source documents are slides, diagrams, or screenshots. This tutorial builds a llamaindex multimodal rag gpt-4o images pipeline that indexes image nodes and answers questions about their visual content using GPT-4o’s vision capabilities.

Prerequisites

Python 3.10 or newer
An OpenAI API key (or any OpenAI-compatible endpoint token)
Familiarity with basic LlamaIndex concepts: indices, query engines, documents

Install the required packages:

pip install llama-index-core llama-index-readers-file \
    llama-index-embeddings-clip llama-index-multi-modal-llms-openai \
    pillow python-dotenv

Create a .env file with your key:

echo "OPENAI_API_KEY=sk-..." > .env

Step 1: Generate sample images

We avoid external downloads by rendering simple PNGs with PIL. This keeps the tutorial fully runnable offline except for the model calls.

from PIL import Image, ImageDraw
import os

os.makedirs("data", exist_ok=True)

img1 = Image.new("RGB", (400, 200), "white")
ImageDraw.Draw(img1).text((10, 80), "Quarterly Revenue: Q1 120k, Q2 150k", fill="black")
img1.save("data/revenue.png")

img2 = Image.new("RGB", (400, 200), "white")
ImageDraw.Draw(img2).text((10, 80), "System Architecture: API -> Cache -> DB", fill="black")
img2.save("data/arch.png")

You now have two image documents with embedded text. In a real deployment these would be scanned reports, UI screenshots, or whiteboard photos.

Step 2: Load image documents

LlamaIndex’s SimpleDirectoryReader detects image files and loads them as Document objects with the image binary attached as a metadata field.

from llama_index.core import SimpleDirectoryReader

docs = SimpleDirectoryReader(
    "data",
    required_exts=[".png"]
).load_data()

print(f"Loaded {len(docs)} documents")
# Loaded 2 documents

Step 3: Embed with CLIP and build the index

Text and images need a shared embedding space. ClipEmbedding projects both modalities into the same vector space, enabling similarity search across images using text queries.

from llama_index.core import VectorStoreIndex
from llama_index.embeddings.clip import ClipEmbedding

embed_model = ClipEmbedding()

index = VectorStoreIndex.from_documents(
    docs,
    embed_model=embed_model,
    show_progress=True
)

How CLIP alignment works

CLIP was trained on image–caption pairs, so its encoder maps a PNG of a bar chart and the phrase “quarterly revenue chart” to nearby vectors. That is what lets a pure-text query retrieve a relevant image node without any OCR preprocessing. The index stores these vectors in an in-memory store by default; swap in Chroma or Pinecone for persistence.

Step 4: Query with GPT-4o multimodal LLM

Attach a multimodal LLM to the query engine. OpenAIMultiModal sends the retrieved image nodes to GPT-4o alongside the prompt using the vision chat endpoint.

from llama_index.multi_modal_llms.openai import OpenAIMultiModal
import os
from dotenv import load_dotenv

load_dotenv()
mm_llm = OpenAIMultiModal(
    model="gpt-4o",
    api_key=os.environ["OPENAI_API_KEY"],
    max_new_tokens=512
)

query_engine = index.as_query_engine(
    multi_modal_llm=mm_llm,
    similarity_top_k=1
)

response = query_engine.query("What was the revenue in Q2?")
print(str(response))

Inspecting retrieved nodes

Before trusting the answer, confirm which image was pulled:

retriever = index.as_retriever(similarity_top_k=1)
nodes = retriever.retrieve("What was the revenue in Q2?")
print(nodes[0].metadata["file_name"])
# revenue.png

Expected output at query time

GPT-4o receives the top retrieved image (revenue.png) and answers from its visual content:

The revenue in Q2 was 150k.

If you ask about the second image:

response = query_engine.query("What sits between the API and the DB?")
print(str(response))

Output:

The cache sits between the API and the DB in the system architecture.

This confirms the llamaindex multimodal rag gpt-4o images flow retrieves the correct image and grounds the answer in its pixels rather than parametric memory.

Step 5: Mix text and image sources

Real corpora rarely contain only images. Add a text file to the same directory and re-index.

with open("data/notes.txt", "w") as f:
    f.write("On-call rotation: Alice (Mon), Bob (Tue). Escalation after 15m.")

docs = SimpleDirectoryReader("data", required_exts=[".png", ".txt"]).load_data()
index = VectorStoreIndex.from_documents(docs, embed_model=embed_model)

CLIP embeds the text file as well, so a query like “Who is on-call Tuesday?” may retrieve the text node, while “Show the architecture” retrieves the image. The same multi_modal_llm handles both; for pure-text nodes GPT-4o simply receives no image content blocks.

Step 6: Streamline retrieval with metadata filters

If your image store grows, filter by filename to reduce noise:

from llama_index.core.vector_stores import MetadataFilters, FilterCondition

filters = MetadataFilters.from_dicts(
    [{"key": "file_name", "value": "arch.png", "operator": "=="}],
    condition=FilterCondition.AND
)

query_engine = index.as_query_engine(
    multi_modal_llm=mm_llm,
    similarity_top_k=1,
    filters=filters
)

This forces the retriever to only consider arch.png, useful when the calling code already knows the document scope.

Step 7: Send multiple images per query

GPT-4o accepts several images in one turn. Raise similarity_top_k to include context from both files:

query_engine = index.as_query_engine(
    multi_modal_llm=mm_llm,
    similarity_top_k=2
)
response = query_engine.query("Compare the revenue chart and the architecture diagram")

The synthesizer passes both retrieved nodes as separate image blocks. Keep top_k small; CLIP recall drops as the vector space gets crowded with dissimilar assets.

Production considerations

The code above calls OpenAI directly. In production, provider outages or rate limits will break the pipeline. You can point OpenAIMultiModal at any OpenAI-compatible base URL without changing the LlamaIndex logic:

mm_llm = OpenAIMultiModal(
    model="gpt-4o",
    api_key=os.environ["GATEWAY_KEY"],
    api_base="https://api.n4n.ai/v1"  # OpenAI-compatible, 240+ models, auto fallback
)

A gateway like n4n.ai fronts multiple providers, automatically failing over when OpenAI is degraded, and forwards cache-control hints so provider-side prompt caching works. Per-token metering gives you cost visibility per request. The rest of your llamaindex multimodal rag gpt-4o images stack stays identical.

Troubleshooting

ValueError: No documents loaded — check required_exts matches your files.
AuthenticationError — confirm .env is loaded before constructing the LLM.
Blurry answers — increase similarity_top_k or verify the image text is legible; CLIP retrieves on semantics, not OCR confidence.

Cleanup

Delete the data directory and the .env file when done. The index lives in memory; for persistence, use a StorageContext:

index.storage_context.persist(persist_dir="./storage")

That is the full loop: render images, embed with CLIP, retrieve with vector search, and synthesize answers with GPT-4o vision. Point it at a folder of PDF figures or product screenshots and you have a grounded multimodal assistant that cites pixels instead of guessing.