When you stand up a RAG system with LlamaIndex, the two heaviest API surfaces are not the same thing. The debate of llamaindex embeddings vs chat completions comes down to two separate endpoints with disjoint responsibilities: one turns text into vectors, the other turns prompts into generated tokens. Treating them as interchangeable will burn your budget and wreck your latency budget.
Capabilities: what each endpoint actually does
The embeddings endpoint accepts a list of strings and returns a fixed-dimensional vector per string. Those vectors are the only thing a vector database can meaningfully compare. In LlamaIndex, this happens during from_documents or when you call insert on an existing index. No language understanding beyond semantic proximity is performed server-side.
The chat completions endpoint accepts a message list and returns either a single completion or a stream of tokens. This is where reasoning, summarization, and tool calling live. LlamaIndex uses it at query time inside as_query_engine, ChatEngine, or any ResponseSynthesizer.
You cannot substitute one for the other. A vector is not a sentence, and a completion cannot be indexed without another embeddings call.
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
llm = OpenAI(model="gpt-4o-mini")
docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(docs, embed_model=embed_model)
query_engine = index.as_query_engine(llm=llm)That snippet shows the split clearly: two distinct client classes, two model names, two bills.
Cost model
Embeddings are priced on input tokens only. There is no output token count because the model emits a fixed-size array, not a variable-length sequence. For OpenAI’s text-embedding-3-small, that cost is a fraction of a cent per million tokens—cheap enough that most teams never notice it.
Chat completions bill input and output tokens. Output is the killer: a query that retrieves 10 chunks of 500 tokens each and asks for a 300-token answer can easily consume 5,000+ input tokens and 300 output tokens per call. Multiply by daily query volume and the embeddings cost becomes rounding error.
The llamaindex embeddings vs chat completions cost asymmetry means you should optimize embeddings model choice for dimension/quality, not price, and hammer chat completions with caching and smaller models where possible.
Latency and throughput
Embeddings calls are embarrassingly parallel. Providers accept batches of up to hundreds of texts per request, and GPU utilization stays high because matrix multiplies dominate. Typical wall-clock for a batch of 100 chunks is sub-second on most hosted APIs.
Chat completions latency splits into time-to-first-token (TTFT) and generation throughput (tokens/sec). TTFT includes prompt processing of your retrieved context; generation is bounded by the model’s decode speed. Streaming helps perceived latency but not total compute.
Throughput math is different: embeddings give you vectors for many documents in one request; chat completions process one conversation at a time. If you need to embed 1M documents, do it offline in batches. If you need to serve 1k concurrent users, your chat endpoint autoscaling matters far more.
Ergonomics in LlamaIndex
LlamaIndex forces the separation early. You pass embed_model to index constructors and llm to query engines. They can point at different providers, different API keys, different base_urls.
from llama_index.core import Settings
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-large")
Settings.llm = OpenAI(model="gpt-4o")Setting them globally via Settings is convenient but hides the boundary. For production, inject explicitly per index and per query engine so you can swap the chat model for a cheaper one during off-peak without touching indexing code.
The raw HTTP shapes also differ. Embeddings POST to /v1/embeddings with {"input": [...], "model": "..."}. Chat POSTs to /v1/chat/completions with {"messages": [...], "model": "..."}. Both are OpenAI-compatible, which means a single gateway can front them—if you route both through a single OpenAI-compatible gateway like n4n.ai, you get access to 240+ models with automatic fallback when a provider is rate-limited, but the LlamaIndex abstraction still treats them as two objects.
Ecosystem and model availability
Embedding model choice is narrow. OpenAI ships three sizes, Cohere has a few, open-source BGE or E5 models run locally. The vector space is not interoperable across vendors, so you pick one and commit.
Chat completions has hundreds of options: GPT-4o, Claude, Llama-3, Mistral, and a long tail of fine-tunes. LlamaIndex’s llms module abstracts them uniformly, but each has different context windows, tool-calling support, and system prompt behavior.
This imbalance means your embeddings decision is sticky; your chat model decision should be fluid. The llamaindex embeddings vs chat completions contrast is most acute here: one is infrastructure, the other is a tunable runtime parameter.
Limits and quotas
Embeddings endpoints enforce a max input token count per text (often 8,191 for OpenAI) and a max batch size. Exceed either and the request 400s. You must chunk documents before embedding—LlamaIndex’s TokenTextSplitter handles this.
Chat completions enforce a context window (e.g., 128k tokens) and per-minute token quotas that are usually tighter than embeddings quotas. When the chat model is degraded, users see errors; when embeddings are degraded, your offline index job stalls.
Rate limits are tracked separately by endpoint. A 500k embeddings/day quota does not help when your chat completions quota is 10k tokens/min and you’re serving live traffic.
Head-to-head summary
| Dimension | Embeddings endpoint | Chat completions endpoint |
|---|---|---|
| Primary job | Convert text to fixed-dim vectors | Generate text from prompt |
| Billing | Input tokens only | Input + output tokens |
| Typical latency | 10–100ms per batch (provider dependent) | 100ms+ TTFT plus generation time |
| Batch efficiency | High (many texts per call) | Low (one sequence per call, streaming helps) |
| LlamaIndex class | OpenAIEmbedding / HuggingFaceEmbedding |
OpenAI / Ollama / ChatEngine |
| Model variety | Few stable options per vendor | Hundreds across vendors |
| Rate limits | Separate quota, often higher req/s | Separate quota, often tighter |
| Failure mode | Index silently degrades or job stalls | User-facing timeout or fallback |
Which to choose: verdict by use case
Offline document ingestion. Use the embeddings endpoint exclusively. Batch aggressively, pick the smallest dimension that preserves recall, and run it as a background job. Chat completions are irrelevant here.
Low-latency Q&A over known docs. Chat completions are the bottleneck. Use a small chat model (gpt-4o-mini, llama-3-8b) with a strong embeddings model. Cache completion responses for identical retrieved contexts.
High-throughput agentic pipelines. You need both, but treat them as separate services. Scale embeddings horizontally for any re-indexing; put chat behind a queue with fallback models. The llamaindex embeddings vs chat completions split lets you swap a degraded chat provider without re-embedding a single byte.
Cost-sensitive prototyping. Default to text-embedding-3-small and a mini chat model. Measure the retrieval quality before upgrading embeddings dimension; measure answer quality before upgrading chat tier.
Multi-provider redundancy. Point both LlamaIndex components at one OpenAI-compatible base URL that fronts multiple providers. Keep the code split—embeddings and chat remain distinct objects—but let the gateway handle routing and cache-control hints.
The separation is not accidental. Embeddings and chat completions solve different problems, bill differently, and fail differently. Architect for that split from day one.