LlamaIndex LLM vs embedding model setup on n4n.ai

If you’re following a llamaindex llm vs embedding model n4n.ai tutorial, the key insight is that LlamaIndex treats these as two distinct abstractions — llm for generation and embed_model for retrieval — and wiring them correctly determines whether your RAG pipeline works or silently returns garbage. n4n.ai exposes both through a single OpenAI-compatible endpoint, but you still need to configure each model explicitly in your Settings object.

Why the distinction matters

LlamaIndex’s Settings class holds global defaults. When you instantiate a VectorStoreIndex, it uses Settings.embed_model to encode documents and queries. When you call query_engine.query(), it uses Settings.llm to synthesize answers. Swapping one without the other is the most common misconfiguration — you might embed with text-embedding-3-large but generate with gpt-4o-mini, or vice versa, and wonder why latency or quality feels off.

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini", api_base="https://api.n4n.ai/v1", api_key="YOUR_KEY")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-large", api_base="https://api.n4n.ai/v1", api_key="YOUR_KEY")

Both clients point at the same base URL. The gateway routes each request to the appropriate provider based on model name, handles fallback when a provider degrades, and meters tokens per model — so your embedding costs don’t get mixed into generation spend.

Choosing models for each role

Embedding models optimize for semantic similarity at scale. Generation models optimize for reasoning, instruction following, and context window. They have different cost curves, latency profiles, and quality tradeoffs.

Don’t default to the same model family for both. A common mistake: using gpt-4o-mini for embeddings because “it’s cheaper.” It isn’t an embedding model — it won’t produce vectors. Conversely, don’t try to generate with text-embedding-3-large. The gateway will reject the request with a clear error, but you’ll waste a round trip.

Wiring per-component instead of globals

Global Settings work for prototypes. Production systems need per-component control — different embedding models for different indexes, different LLMs for different query engines. Pass instances explicitly:

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Index A: high-recall embeddings, fast generator
embed_a = OpenAIEmbedding(model="voyage-3-large", api_base="https://api.n4n.ai/v1", api_key=KEY)
llm_fast = OpenAI(model="gpt-4o-mini", api_base="https://api.n4n.ai/v1", api_key=KEY)

index_a = VectorStoreIndex.from_documents(docs_a, embed_model=embed_a)
query_engine_a = index_a.as_query_engine(llm=llm_fast)

# Index B: smaller embeddings, stronger generator
embed_b = OpenAIEmbedding(model="text-embedding-3-small", api_base="https://api.n4n.ai/v1", api_key=KEY)
llm_strong = OpenAI(model="claude-3-5-sonnet", api_base="https://api.n4n.ai/v1", api_key=KEY)

index_b = VectorStoreIndex.from_documents(docs_b, embed_model=embed_b)
query_engine_b = index_b.as_query_engine(llm=llm_strong)

This pattern also makes testing easier — swap embed_model in a unit test without touching global state.

Handling dimension mismatches

Each embedding model outputs a fixed vector dimension. text-embedding-3-large produces 3072 dimensions. text-embedding-3-small produces 1536. voyage-3-large produces 1024. Your vector store must match.

If you’re using a managed store (Pinecone, Weaviate, Qdrant), create the index with the correct dimension:

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key=PINECONE_KEY)
pc.create_index(
    name="rag-embeddings",
    dimension=3072,  # must match embed_model output
    metric="cosine",
    spec=ServerlessSpec(cloud="aws", region="us-east-1")
)

If you switch embedding models later, you need a new index — or a migration strategy. LlamaIndex won’t validate this for you. The first insert() will succeed; queries will silently return nonsense because the query vector lives in a different space than the stored vectors.

Streaming and async patterns

Both LLM and embedding clients support streaming and async. Use them.

# Streaming generation
response = query_engine.query("Summarize the Q3 risks")
for token in response.response_gen:
    print(token, end="", flush=True)

# Async embedding (batch documents)
nodes = await embed_model.aget_text_embedding_batch(texts)

The gateway forwards provider-level streaming (SSE for OpenAI-compatible, native for Anthropic). Don’t wrap sync calls in asyncio.to_thread() — use the native async methods. They release the event loop during network I/O and handle backpressure correctly.

Common pitfalls

Pitfall 1: Forgetting api_base on one client.
If you set Settings.llm with the gateway URL but leave Settings.embed_model pointing at OpenAI directly, your embedding traffic bypasses the gateway — no fallback, no unified metering, no cache-control forwarding. Always set both.

Pitfall 2: Mixing model and deployment_name for Azure.
If you route through Azure OpenAI deployments, the parameter is deployment_name, not model. The gateway accepts model and maps it to the correct provider deployment, but the native Azure client behaves differently. Stick to the gateway’s OpenAI-compatible interface unless you have a specific Azure-only requirement.

Pitfall 3: Assuming text-embedding-ada-002 is still the default.
LlamaIndex’s OpenAIEmbedding defaults to text-embedding-ada-002 (1536 dim). If you create an index with that default, then later switch to text-embedding-3-large without recreating the index, you’ll get dimension errors at query time. Be explicit:

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-large",  # explicit, not default
    api_base="https://api.n4n.ai/v1",
    api_key=KEY
)

Pitfall 4: Ignoring token limits on the embedding side.
Embedding models have max input tokens (8191 for text-embedding-3-*, 32000 for voyage-3-large). LlamaIndex’s SentenceSplitter chunks documents before embedding, but the default chunk size (1024) with overlap (200) can still produce chunks that exceed limits for very long tokens-per-word languages. Set chunk_size conservatively:

from llama_index.core.node_parser import SentenceSplitter

parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)
nodes = parser.get_nodes_from_documents(docs)

Cost observability

Because the gateway meters per model, you can attribute spend precisely. Enable response headers to capture usage:

import httpx

client = httpx.Client(base_url="https://api.n4n.ai/v1")
response = client.post(
    "/embeddings",
    json={"model": "text-embedding-3-large", "input": "test"},
    headers={"Authorization": f"Bearer {KEY}"}
)
usage = response.headers.get("x-usage-tokens")  # gateway-injected

Log this alongside your application metrics. Embedding costs scale with document count and chunk size; generation costs scale with query volume and context window. They grow at different rates — track them separately.

Fallback behavior

The gateway automatically fails over when a provider returns 429, 5xx, or exceeds latency thresholds. For embeddings, this is transparent — the request retries on a healthy provider. For generation, streaming responses may switch mid-stream if the primary degrades. Your code doesn’t need to change, but you should:

1. Set reasonable timeouts on the client (30s for embeddings, 120s for generation).
2. Log when fallback occurs (the gateway includes x-provider header on responses).
3. Test failure scenarios — kill a provider in staging and verify latency stays within SLA.

llm = OpenAI(
    model="gpt-4o",
    api_base="https://api.n4n.ai/v1",
    api_key=KEY,
    timeout=120.0,
    max_retries=2  # gateway handles provider-level retry; this is client-level
)

Testing the wiring

Write a smoke test that exercises both paths:

def test_llm_and_embedding_wired():
    # Embedding path
    vec = Settings.embed_model.get_text_embedding("hello world")
    assert len(vec) == 3072  # matches text-embedding-3-large
    
    # Generation path
    resp = Settings.llm.complete("Say 'ok'")
    assert "ok" in resp.text.lower()
    
    # End-to-end
    index = VectorStoreIndex.from_documents([Document(text="Revenue was $10M")])
    qe = index.as_query_engine()
    answer = qe.query("What was revenue?")
    assert "$10M" in str(answer)

Run this in CI against a staging gateway. It catches dimension mismatches, auth issues, and routing errors before they hit production.

Summary checklist

• Set Settings.llm and Settings.embed_model explicitly with api_base="https://api.n4n.ai/v1"
• Choose embedding model for retrieval quality, generation model for reasoning — don’t conflate them
• Match vector store dimension to embedding model output
• Pass embed_model and llm per-component in production, not just globals
• Use async/streaming methods natively
• Set conservative chunk sizes for multilingual content
• Monitor per-model token usage via gateway headers
• Test fallback behavior in staging

The llamaindex llm vs embedding model n4n.ai tutorial pattern is straightforward once you internalize that these are two independent pipelines sharing only a gateway endpoint. Configure each deliberately, observe each separately, and your RAG system will behave predictably under load.