Building a chat engine in LlamaIndex with n4n.ai's API

The fastest way to stand up a production-ready conversational interface is to wire a llamaindex chat engine n4n.ai api integration into your Python service. This tutorial builds a SimpleChatEngine first, then extends it to a context-aware engine backed by a vector index, using nothing but LlamaIndex’s standard OpenAI-compatible client. You will see runnable code, expected outputs, and the few knobs that actually matter in production.

Prerequisites

• Python 3.10 or newer
• llama-index and llama-index-llms-openai (LlamaIndex v0.10+)
• A valid API key for the n4n.ai gateway (it uses OpenAI-style bearer auth)
• A local folder data/ with a few .txt files if you want to run the RAG portion
• python-dotenv to keep secrets out of source

Install the dependencies:

pip install llama-index llama-index-llms-openai python-dotenv

Create a .env file:

N4N_API_KEY=sk-your-key-here

Configure the LLM client

Because the gateway speaks the OpenAI protocol, we instantiate LlamaIndex’s OpenAI class and point api_base at the endpoint. Model names follow the gateway’s routing scheme; you can address any of the 240+ available models by using the provider-prefixed string.

import os
from dotenv import load_dotenv
from llama_index.llms.openai import OpenAI

load_dotenv()

llm = OpenAI(
    model="openai/gpt-4o-mini",
    api_key=os.getenv("N4N_API_KEY"),
    api_base="https://api.n4n.ai/v1",
    temperature=0.2,
    max_tokens=512,
)

# Fail fast if misconfigured
assert llm.api_key, "N4N_API_KEY not loaded"

If the key is missing or the base URL is wrong, the first request raises an auth or connection error. Do this check at process startup, not in the request path.

Step 1: Stateless chat

A SimpleChatEngine wraps the LLM with no retrieval. It is useful for sanity checks and lightweight bots where conversation history is managed elsewhere.

from llama_index.core.chat_engine import SimpleChatEngine

chat_engine = SimpleChatEngine.from_defaults(llm=llm)
resp = chat_engine.chat("Explain vector search in one sentence.")
print(resp)

Expected output:

Vector search finds items by nearest neighbor distance in embedding space instead of lexical matching.

This engine has no memory. Ask a follow-up and it will not remember the prior turn.

Step 2: Add conversation memory

Without memory, the engine forgets prior turns. Use ChatMemoryBuffer to keep a rolling window bounded by token count. The buffer is passed into the engine constructor.

from llama_index.core.memory import ChatMemoryBuffer

memory = ChatMemoryBuffer(token_limit=2000)
chat_engine = SimpleChatEngine.from_defaults(llm=llm, memory=memory)

chat_engine.chat("My name is Ada.")
followup = chat_engine.chat("What's my name?")
print(followup)

Expected output:

Your name is Ada.

The buffer truncates oldest messages when the token limit is exceeded. Set the limit based on your model’s context window minus headroom for the response and any retrieved context. A 2000-token memory is safe for 8k models with small RAG payloads.

Step 3: Context-aware chat with a vector index

For document-grounded answers, build a VectorStoreIndex and use the context chat mode. This runs retrieval per turn and injects the top nodes into the prompt.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

docs = SimpleDirectoryReader("data/").load_data()
index = VectorStoreIndex.from_documents(docs)

chat_engine = index.as_chat_engine(
    chat_mode="context",
    llm=llm,
    memory=memory,
)
response = chat_engine.chat("Summarize the onboarding doc.")
print(response.response)

If data/ contains onboarding.txt with three paragraphs about account setup, you might see:

The onboarding doc outlines account creation, API key generation, and the first test request against the gateway.

Tuning retrieval

The default similarity_top_k is 2. For denser docs, raise it, but watch your token budget:

retriever = index.as_retriever(similarity_top_k=4)
chat_engine = index.as_chat_engine(
    chat_mode="context",
    retriever=retriever,
    llm=llm,
    memory=memory,
)

If you need metadata filtering, attach it to the retriever via vector_store_query_mode or a custom retriever. LlamaIndex does not re-rank by default; add a TransformComponent if precision matters.

Step 4: Stream tokens

Users expect incremental rendering. LlamaIndex exposes stream_chat on every chat engine:

stream = chat_engine.stream_chat("List three risks of self-hosting LLMs.")
for delta in stream.response_gen:
    print(delta, end="", flush=True)

You get token-by-token output without changing the index or memory setup. In a FastAPI app, return a StreamingResponse around response_gen.

Step 5: Async for concurrency

Sync calls block the event loop. If you serve this behind an async web framework, use the achat / astream_chat methods:

import asyncio
from llama_index.core.chat_engine import SimpleChatEngine

async def main():
    engine = SimpleChatEngine.from_defaults(llm=llm)
    result = await engine.achat("Ping")
    print(result)

asyncio.run(main())

The same memory and index objects are safe to share across coroutines only if you serialize access; otherwise create a per-request engine from the shared index. Index objects are cheap to query, but the chat memory is stateful.

Common pitfalls

• Forgot api_base: The client silently calls api.openai.com. Your key will be rejected and you’ll waste an hour.
• Memory not passed: index.as_chat_engine(chat_mode="context") without memory= is stateless. The conversation will feel amnesiac.
• Token math ignored: ChatMemoryBuffer(token_limit=2000) plus similarity_top_k=6 with 500-token chunks equals 5000+ tokens before the response. Truncation kicks in mid-conversation.
• Model string mismatch: The gateway routes on provider/model. Using gpt-4o-mini instead of openai/gpt-4o-mini may hit a default provider you didn’t intend.

Production notes

When you ship the llamaindex chat engine n4n.ai api integration, the gateway’s automatic fallback shields you from individual provider rate limits—no custom retry loop required. Set temperature low (0.1–0.3) for factual RAG, and cache static system prompts by forwarding provider cache-control hints through the client if your latency budget is tight.

Keep memory limits conservative; a 2000-token buffer plus retrieved context can still blow an 8k window if you retrieve aggressively. Measure with real conversations before scaling top_k. The chat engine code itself is stable; the operational risk is almost always in token accounting and model routing, not in LlamaIndex.

You now have a runnable chat engine that talks to an OpenAI-compatible gateway, supports memory, RAG, streaming, and async—without writing a single line of HTTP client code.