LlamaIndex TypeScript setup with n4n.ai

LlamaIndex TypeScript has matured significantly since its 0.1 release, and pairing it with an OpenAI-compatible gateway like n4n.ai gives you model flexibility without rewriting your inference layer. This tutorial walks through a complete, runnable RAG pipeline from empty directory to querying your own documents.

Prerequisites

• Node.js 20+ (LTS recommended)
• npm 10+ or pnpm/yarn equivalent
• An n4n.ai API key (or any OpenAI-compatible endpoint)
• Basic familiarity with TypeScript and async/await

Verify your environment:

node --version
# v20.12.0 or higher

npm --version
# 10.5.0 or higher

Initialize the project

Create a fresh directory and initialize a TypeScript project with strict settings:

mkdir llamaindex-rag && cd llamaindex-rag
npm init -y
npm install typescript tsx @types/node --save-dev

Create tsconfig.json with strict mode enabled — this catches real bugs early:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "lib": ["ES2022"],
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": true,
    "outDir": "./dist",
    "rootDir": "./src"
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "dist"]
}

Add a package.json script for running TypeScript directly with tsx:

{
  "scripts": {
    "dev": "tsx watch src/index.ts",
    "build": "tsc",
    "start": "node dist/index.js"
  }
}

Install LlamaIndex dependencies

LlamaIndex TypeScript splits core functionality across several packages. Install the essentials:

npm install llamaindex @llamaindex/env node-fetch

• llamaindex — core abstractions, vector stores, indices, query engines
• @llamaindex/env — environment configuration (API keys, base URLs)
• node-fetch — required for Node 20+ fetch polyfill in some environments

Configure the gateway

Create src/config.ts to centralize environment configuration. This is where you point LlamaIndex at the n4n.ai endpoint:

// src/config.ts
import { setGlobalConfig } from "llamaindex";
import { env } from "@llamaindex/env";

export function configureGateway() {
  const apiKey = process.env.N4N_API_KEY;
  const baseURL = process.env.N4N_BASE_URL ?? "https://api.n4n.ai/v1";

  if (!apiKey) {
    throw new Error("N4N_API_KEY environment variable is required");
  }

  setGlobalConfig({
    llm: {
      model: "meta-llama/llama-3.1-70b-instruct",
      apiKey,
      baseURL,
      temperature: 0.1,
      maxTokens: 1024,
    },
    embedModel: {
      model: "text-embedding-3-small",
      apiKey,
      baseURL,
      dimensions: 1536,
    },
  });
}

Key points:

• The baseURL points to the OpenAI-compatible endpoint. n4n.ai exposes /v1 at https://api.n4n.ai/v1.
• model for the LLM can be any of the 240+ models available through the gateway — here we use Llama 3.1 70B Instruct.
• The embedding model must match the dimensions your vector store expects. text-embedding-3-small at 1536 dimensions is a safe default.

Create a .env file (never commit this):

# .env
N4N_API_KEY=your-api-key-here
N4N_BASE_URL=https://api.n4n.ai/v1

Load it in your entry point:

// src/index.ts
import "dotenv/config";
import { configureGateway } from "./config";

configureGateway();
console.log("Gateway configured");

Run it to verify:

npm run dev
# Gateway configured

Ingest documents

LlamaIndex’s SimpleDirectoryReader handles PDF, Markdown, text, and more. Create a data directory with sample content:

mkdir data
cat > data/overview.md << 'EOF'
# n4n.ai Architecture Overview

n4n.ai is an LLM inference gateway that provides a single OpenAI-compatible
endpoint addressing 240+ models across multiple providers.

## Key Features

- **Automatic fallback**: When a provider is rate-limited or degraded, requests
  route to healthy alternatives without client changes.
- **Per-token metering**: Usage is tracked per model and provider for accurate
  cost allocation.
- **Cache-control hints**: Provider-level caching directives are forwarded to
  clients, enabling conditional requests and reduced latency.
- **Routing directives**: Clients can specify model preferences, cost ceilings,
  or latency targets via standard headers.

## Supported Providers

The gateway aggregates capacity from Together AI, Fireworks AI, Anyscale,
DeepInfra, and others. Model availability updates automatically.
EOF

Now build the ingestion pipeline in src/ingest.ts:

// src/ingest.ts
import { SimpleDirectoryReader, VectorStoreIndex, storageContextFromDefaults } from "llamaindex";
import { configureGateway } from "./config";

export async function buildIndex() {
  configureGateway();

  // 1. Load documents from the data directory
  const reader = new SimpleDirectoryReader({
    inputDir: "./data",
    recursive: true,
    requiredExts: [".md", ".txt", ".pdf"],
  });

  const documents = await reader.loadData();
  console.log(`Loaded ${documents.length} document(s)`);

  // 2. Create a vector store index with in-memory storage (swap for persistent later)
  const storageContext = await storageContextFromDefaults({
    persistDir: "./storage",
  });

  const index = await VectorStoreIndex.fromDocuments(documents, {
    storageContext,
  });

  // 3. Persist to disk for reuse
  await index.storageContext.persist({ persistDir: "./storage" });
  console.log("Index persisted to ./storage");

  return index;
}

// Allow running directly: npx tsx src/ingest.ts
if (import.meta.url === `file://${process.argv[1]}`) {
  buildIndex().catch(console.error);
}

Run the ingestion:

npx tsx src/ingest.ts
# Loaded 1 document(s)
# Index persisted to ./storage

The ./storage directory now contains your vector index — docstore.json, vector_store.json, and index_store.json.

Query the index

Create src/query.ts with a retrieval-augmented query engine:

// src/query.ts
import { VectorStoreIndex, storageContextFromDefaults, Settings } from "llamaindex";
import { configureGateway } from "./config";

export async function queryIndex(question: string) {
  configureGateway();

  // Load the persisted index
  const storageContext = await storageContextFromDefaults({
    persistDir: "./storage",
  });

  const index = await VectorStoreIndex.init({ storageContext });

  // Create a query engine with sensible defaults
  const queryEngine = index.asQueryEngine({
    similarityTopK: 4,
    responseMode: "compact",
  });

  const response = await queryEngine.query({ query: question });
  return response;
}

// CLI entry point
if (import.meta.url === `file://${process.argv[1]}`) {
  const question = process.argv[2] ?? "What providers does n4n.ai aggregate?";
  queryIndex(question)
    .then((r) => console.log("\nAnswer:", r.response))
    .catch(console.error);
}

Test it:

npx tsx src/query.ts "What providers does n4n.ai aggregate?"

Expected output (abbreviated):

Answer: n4n.ai aggregates capacity from Together AI, Fireworks AI, Anyscale, DeepInfra, and others. Model availability updates automatically.

Try a few more questions:

npx tsx src/query.ts "How does automatic fallback work?"
npx tsx src/query.ts "What caching features are supported?"

Add a streaming response

For production UIs, streaming tokens as they arrive is essential. Update src/query.ts:

// src/query.ts (add to existing exports)
export async function streamQuery(question: string) {
  configureGateway();

  const storageContext = await storageContextFromDefaults({
    persistDir: "./storage",
  });

  const index = await VectorStoreIndex.init({ storageContext });

  const queryEngine = index.asQueryEngine({
    similarityTopK: 4,
    responseMode: "compact",
    streaming: true,
  });

  const response = await queryEngine.query({ query: question });

  for await (const chunk of response) {
    process.stdout.write(chunk);
  }
  console.log(); // final newline
}

Test streaming:

npx tsx -e "
import { streamQuery } from './src/query.js';
streamQuery('Explain routing directives in two sentences').catch(console.error);
"

Output appears token-by-token:

Routing directives let clients specify model preferences, cost ceilings, or latency targets via standard headers. The gateway honors these directives when selecting which provider serves each request.

Swap to a persistent vector store

In-memory storage works for prototypes. For production, use a proper vector database. Here’s how to swap in Qdrant (local Docker) with minimal changes.

Start Qdrant:

docker run -d -p 6333:6333 -p 6334:6334 \
  -v $(pwd)/qdrant_data:/qdrant/storage \
  qdrant/qdrant

Install the Qdrant integration:

npm install @llamaindex/qdrant

Update src/ingest.ts to use QdrantVectorStore:

// src/ingest.ts (replace storageContextFromDefaults import)
import { QdrantVectorStore } from "@llamaindex/qdrant";
import { VectorStoreIndex, storageContextFromDefaults, Settings } from "llamaindex";
// ... existing imports

export async function buildIndex() {
  configureGateway();

  const reader = new SimpleDirectoryReader({
    inputDir: "./data",
    recursive: true,
    requiredExts: [".md", ".txt", ".pdf"],
  });

  const documents = await reader.loadData();
  console.log(`Loaded ${documents.length} document(s)`);

  // Qdrant vector store
  const vectorStore = new QdrantVectorStore({
    url: "http://localhost:6333",
    collectionName: "llamaindex_docs",
    // Optional: enable hybrid search with sparse vectors
    // enableHybrid: true,
  });

  const storageContext = await storageContextFromDefaults({
    vectorStore,
  });

  const index = await VectorStoreIndex.fromDocuments(documents, {
    storageContext,
  });

  console.log("Index built in Qdrant");
  return index;
}

Update src/query.ts similarly:

// src/query.ts (replace storageContextFromDefaults usage)
import { QdrantVectorStore } from "@llamaindex/qdrant";
import { VectorStoreIndex, Settings } from "llamaindex";
// ... existing imports

export async function queryIndex(question: string) {
  configureGateway();

  const vectorStore = new QdrantVectorStore({
    url: "http://localhost:6333",
    collectionName: "llamaindex_docs",
  });

  const index = await VectorStoreIndex.init({ vectorStore });
  // ... rest unchanged
}

Re-ingest and query — same API, now backed by Qdrant:

npx tsx src/ingest.ts
npx tsx src/query.ts "What is per-token metering?"

Handle routing directives in client code

One advantage of an OpenAI-compatible gateway is passing routing hints via headers. LlamaIndex’s OpenAI-compatible LLM class accepts additionalHeaders:

// src/config.ts (extended)
import { setGlobalConfig, OpenAI } from "llamaindex";

export function configureGatewayWithRouting(options?: {
  maxCostPerToken?: number;
  maxLatencyMs?: number;
  preferredProviders?: string[];
}) {
  const apiKey = process.env.N4N_API_KEY;
  const baseURL = process.env.N4N_BASE_URL ?? "https://api.n4n.ai/v1";

  if (!apiKey) {
    throw new Error("N4N_API_KEY environment variable is required");
  }

  const headers: Record<string, string> = {};

  if (options?.maxCostPerToken) {
    headers["x-max-cost-per-token"] = String(options.maxCostPerToken);
  }
  if (options?.maxLatencyMs) {
    headers["x-max-latency-ms"] = String(options.maxLatencyMs);
  }
  if (options?.preferredProviders?.length) {
    headers["x-preferred-providers"] = options.preferredProviders.join(",");
  }

  setGlobalConfig({
    llm: new OpenAI({
      model: "meta-llama/llama-3.1-70b-instruct",
      apiKey,
      baseURL,
      temperature: 0.1,
      maxTokens: 1024,
      additionalHeaders: headers,
    }),
    embedModel: {
      model: "text-embedding-3-small",
      apiKey,
      baseURL,
      dimensions: 1536,
    },
  });
}

Usage:

// Cost-conscious query: prefer cheaper providers, cap at $0.50/M tokens
configureGatewayWithRouting({ maxCostPerToken: 0.0000005 });

// Latency-sensitive: cap at 500ms, prefer Fireworks and Together
configureGatewayWithRouting({
  maxLatencyMs: 500,
  preferredProviders: ["fireworks", "together"],
});

The gateway honors these directives when selecting which upstream provider serves the request.

Error handling and retries

Production code needs resilient error handling. Wrap queries with exponential backoff:

// src/retry.ts
export async function withRetry<T>(
  fn: () => Promise<T>,
  options: { maxAttempts?: number; baseDelayMs?: number } = {}
): Promise<T> {
  const { maxAttempts = 3, baseDelayMs = 1000 } = options;
  let lastError: Error;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error as Error;
      const isRetryable =
        error instanceof Response && [429, 500, 502, 503, 504].includes(error.status);

      if (!isRetryable || attempt === maxAttempts) {
        throw error;
      }

      const delay = baseDelayMs * Math.pow(2, attempt - 1) + Math.random() * 200;
      console.warn(`Attempt ${attempt} failed, retrying in ${delay.toFixed(0)}ms:`, error);
      await new Promise((r) => setTimeout(r, delay));
    }
  }

  throw lastError!;
}

Apply it in query.ts:

import { withRetry } from "./retry";

export async function queryIndex(question: string) {
  return withRetry(async () => {
    // ... existing query logic
  });
}

Add a simple API server

Expose the query engine via HTTP for frontend integration. Install Fastify:

npm install fastify @fastify/cors
npm install @types/node --save-dev  # if not already present

Create src/server.ts:

// src/server.ts
import Fastify from "fastify";
import cors from "@fastify/cors";
import { queryIndex, streamQuery } from "./query";
import { configureGateway } from "./config";

const app = Fastify({ logger: true });
await app.register(cors, { origin: true });

app.get("/health", async () => ({ status: "ok" }));

app.post("/query", async (request, reply) => {
  const { question, stream } = request.body as { question: string; stream?: boolean };

  if (!question) {
    return reply.code(400).send({ error: "question is required" });
  }

  configureGateway();

  if (stream) {
    reply.raw.writeHead(200, {
      "Content-Type": "text/plain; charset=utf-8",
      "Transfer-Encoding": "chunked",
    });

    const storageContext = await (await import("llamaindex")).storageContextFromDefaults({
      persistDir: "./storage",
    });
    const { VectorStoreIndex } = await import("llamaindex");
    const index = await VectorStoreIndex.init({ storageContext });
    const queryEngine = index.asQueryEngine({ similarityTopK: 4, streaming: true });
    const response = await queryEngine.query({ query: question });

    for await (const chunk of response) {
      reply.raw.write(chunk);
    }
    reply.raw.end();
    return reply;
  }

  const response = await queryIndex(question);
  return { answer: response.response, sources: response.sourceNodes?.map((n) => n.nodeId) };
});

const port = Number(process.env.PORT) ?? 3000;
await app.listen({ port, host: "0.0.0.0" });
console.log(`Server listening on http://localhost:${port}`);

Run it:

npx tsx src/server.ts

Test with curl:

curl -X POST http://localhost:3000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is automatic fallback?"}'

# Streaming:
curl -N -X POST http://localhost:3000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is automatic fallback?", "stream": true}'

Project structure recap

llamaindex-rag/
├── data/
│   └── overview.md
├── src/
│   ├── config.ts          # Gateway configuration
│   ├── ingest.ts          # Document ingestion pipeline
│   ├── query.ts           # Query engine + streaming
│   ├── retry.ts           # Exponential backoff helper
│   ├── server.ts          # Fastify API server
│   └── index.ts           # Entry point (loads .env)
├── storage/               # Persisted vector index (in-memory default)
├── qdrant_data/           # Qdrant persistence (if using Docker)
├── .env                   # API keys (gitignored)
├── package.json
└── tsconfig.json

Next steps for production

1. Authentication: Add API key validation middleware to /query.
2. Observability: Integrate OpenTelemetry; LlamaIndex emits spans for retrieval and generation.
3. Evaluation: Use llamaindex/evaluation to measure faithfulness and relevance against a golden set.
4. Hybrid search: Enable sparse+dense retrieval in Qdrant for keyword-heavy queries.
5. Multi-tenancy: Partition collections by tenant ID in the vector store.
6. Cache invalidation: Implement a document update pipeline that re-embeds changed files.

Summary

You now have a complete LlamaIndex TypeScript RAG pipeline:

• Ingestion via SimpleDirectoryReader → VectorStoreIndex
• Persistence to local JSON or Qdrant
• Querying with configurable top-K, compact responses, and streaming
• Routing control via gateway headers for cost/latency tradeoffs
• HTTP API ready for frontend integration

The gateway abstraction means you can swap models — Llama 3.1 70B, Mixtral, Qwen, or any of the 240+ available — without changing application code. That flexibility is the point.