Cache embeddings in LlamaIndex to reduce OpenAI API spend

If you’re building retrieval-augmented generation systems, you’ll burn through embedding tokens re-processing the same chunks on every index rebuild. This tutorial shows how a llamaindex embedding cache reduce openai costs by persisting vectors locally or in Redis, so repeated calls to the same text skip the API entirely. We’ll build a working example from scratch, measure the difference, and cover the production details that matter.

Prerequisites

• Python 3.10 or newer
• pip install llama-index llama-index-embeddings-openai redis
• An OpenAI API key (or any OpenAI-compatible endpoint) exported as OPENAI_API_KEY
• Docker (optional, for Redis): docker run -d -p 6379:6379 redis:7

We’ll use text-embedding-3-small as the model. The patterns apply to any embedding model LlamaIndex supports.

Step 1: Baseline — No Cache

First, confirm the naive behavior. The script below embeds a short list of documents twice. Every call hits the network.

import os
from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_key=os.environ["OPENAI_API_KEY"],
)

texts = ["LlamaIndex simplifies RAG", "Caching embeddings cuts cost"]

# First pass
vectors1 = embed_model.get_text_embedding_batch(texts)
print(f"First pass: {len(vectors1)} vectors, dim={len(vectors1[0])}")

# Second pass (identical input)
vectors2 = embed_model.get_text_embedding_batch(texts)
print(f"Second pass: {len(vectors2)} vectors, dim={len(vectors2[0])}")

Expected output:

First pass: 2 vectors, dim=1536
Second pass: 2 vectors, dim=1536

Both passes triggered two API requests. At scale—thousands of chunks across repeated index builds—this doubles your embedding spend for zero new information.

Step 2: Add an In-Memory Cache for Development

LlamaIndex ships a simple InMemoryCache that stores embeddings in a Python dict. It’s perfect for notebooks and single-process scripts.

from llama_index.core.cache import InMemoryCache

embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    cache=InMemoryCache(),
)

# First call populates cache
v1 = embed_model.get_text_embedding_batch(texts)
# Second call reads from cache (no API request)
v2 = embed_model.get_text_embedding_batch(texts)

assert v1 == v2
print("In-memory cache works: vectors identical, second call served locally")

The cache key is a hash of the text content plus the model name. Changing the model automatically invalidates old entries, so you never mismatch dimensions.

For a single process this is enough to make repeated queries or re-indexing free. But the cache evaporates when the process exits, and it isn’t shared across workers.

Step 3: Redis Cache for Shared, Persistent Storage

In production you typically run multiple workers or rebuild indexes in separate jobs. Use RedisCache to give every process a common embedding store.

from llama_index.core.cache import RedisCache

redis_cache = RedisCache(
    redis_uri="redis://localhost:6379",
    namespace="my_rag_app",
)

embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    cache=redis_cache,
)

# Warm the cache
_ = embed_model.get_text_embedding_batch(texts)

After the first run, inspect Redis:

redis-cli DBSIZE
# => (integer) 2
redis-cli KEYS '*'
# => 1) "my_rag_app:8f3a..."
#    2) "my_rag_app:1c9b..."

Each key maps to a serialized blob of the embedding vector. On the next invocation—even in a different Python process—the entries are fetched from Redis. The OpenAI API is never called for those strings again.

Step 4: Verify Cache Hits Programmatically

Relying on redis-cli is fine, but you should assert cache behavior in tests. Subclass RedisCache to log hits:

class LoggingRedisCache(RedisCache):
    def get(self, key: str):
        val = super().get(key)
        if val is None:
            print(f"MISS  {key[:12]}")
        else:
            print(f"HIT   {key[:12]}")
        return val

embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    cache=LoggingRedisCache(redis_uri="redis://localhost:6379"),
)

embed_model.get_text_embedding_batch(texts)  # MISS x2
embed_model.get_text_embedding_batch(texts)  # HIT x2

Expected output:

MISS  8f3a2c1b9d3a
MISS  1c9b4e7f0a2c
HIT   8f3a2c1b9d3a
HIT   1c9b4e7f0a2c

This confirms the llamaindex embedding cache reduce openai costs by short-circuiting redundant network calls.

Step 5: Cache Invalidation and TTL

Embeddings are deterministic for a given model version, so you rarely need to invalidate. Two cases require action:

1. You switch embedding models (e.g., text-embedding-3-small → text-embedding-3-large). LlamaIndex’s key includes the model name, so old entries are ignored automatically.
2. You suspect provider behavior changed (rare). Use a Redis namespace bump: namespace="v2" to start fresh.

If you want time-based expiry, set it at the Redis level:

redis-cli EXPIRE my_rag_app:8f3a2c1b9d3a 86400

Or configure a max-memory policy (allkeys-lru) in redis.conf so the cache evicts least-recently-used vectors under pressure.

Step 6: Using a Gateway Endpoint

If you route embeddings through an OpenAI-compatible gateway such as n4n.ai, the per-token usage metering still counts every upstream embedding request. Client-side caching in LlamaIndex is therefore the only way to avoid those charges—the gateway cannot infer that a vector was previously computed. Set the api_base and api_key accordingly:

embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_base="https://api.n4n.ai/v1",
    api_key=os.environ["N4N_API_KEY"],
    cache=RedisCache(redis_uri="redis://localhost:6379"),
)

The cache layer sits in front of the HTTP client, so the gateway never sees duplicated texts.

Step 7: Integrating With a LlamaIndex Index

Finally, wire the cached embed model into Settings so any index uses it:

from llama_index.core import Settings, VectorStoreIndex, Document

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    cache=RedisCache(redis_uri="redis://localhost:6379"),
)

docs = [Document(text=t) for t in texts]
index = VectorStoreIndex.from_documents(docs)  # embeds + caches

# Rebuilding with same docs hits cache
index2 = VectorStoreIndex.from_documents(docs)
print("Second index build used cached embeddings")

This is the payoff: CI jobs, local experiments, and production re-indexes all share one vector store, and your OpenAI bill reflects only net-new text.

Caveats

• Cache keys are based on exact text. Minor whitespace changes create new entries. Normalize inputs (.strip()) before embedding if you expect near-duplicates.
• InMemoryCache is not thread-safe for concurrent writes in very old LlamaIndex versions; Redis handles concurrency natively.
• Embedding API pricing is per-token of input; caching saves the most when your corpus has heavy overlap across builds.

Implementing a llamaindex embedding cache reduce openai costs with a few lines of configuration. Start with InMemoryCache in dev, move to RedisCache when you scale, and treat the cache as a first-class infrastructure component.