Building a RAG chatbot with Next.js and the Vercel AI SDK

Building a next.js rag chatbot vercel ai sdk means wiring retrieval into a streaming chat route without fighting the framework. This guide builds a working App Router app that embeds your documents, fetches relevant chunks, and streams answers with the Vercel AI SDK. No LangChain, no custom servers—just the SDK and a few lines of vector math.

Prerequisites

• Node.js 18.18+ and npm 9+
• A Next.js 14+ project using the App Router
• An API key for an OpenAI-compatible chat + embeddings model
• Familiarity with React Server Components and TypeScript

If you want provider redundancy later, an OpenAI-compatible gateway key works unchanged with the SDK.

Scaffold the project

Create a fresh app with the App Router and source directory:

npx create-next-app@latest rag-chat --ts --app --eslint --tailwind --src-dir --import-alias "@/*"
cd rag-chat
npm install ai @ai-sdk/openai @ai-sdk/react zod

The ai package provides streamText, embed, and embedMany. @ai-sdk/openai adapts OpenAI (or any compatible endpoint) to the SDK’s interface.

Design the RAG pipeline

Retrieval-augmented generation has three moving parts:

1. Indexing: split source text, embed each chunk, store vectors.
2. Retrieval: embed the user query, rank chunks by cosine similarity.
3. Generation: inject top chunks into the system prompt and stream the model response.

For a tutorial, an in-memory index is enough. In production you’d swap the array for Pinecone, pgvector, or a local HNSW index.

Embed and index documents

Create src/lib/rag.ts. We’ll hardcode a few docs and embed them at module load. text-embedding-3-small returns 1536-dimensional vectors.

import { embedMany, embed } from 'ai';
import { openai } from '@ai-sdk/openai';

const SOURCE_DOCS = [
  'The Vercel AI SDK provides a unified API for streaming text from LLMs.',
  'App Router server actions can securely call model providers without exposing keys.',
  'Cosine similarity measures the angle between two vectors, ignoring magnitude.',
  'RAG reduces hallucination by grounding responses in retrieved context.',
];

export type Chunk = { text: string; vector: number[] };

let indexCache: Chunk[] | null = null;

export async function buildIndex(): Promise<Chunk[]> {
  if (indexCache) return indexCache;
  const { embeddings } = await embedMany({
    model: openai.embedding('text-embedding-3-small'),
    values: SOURCE_DOCS,
  });
  indexCache = SOURCE_DOCS.map((text, i) => ({ text, vector: embeddings[i] }));
  return indexCache;
}

Retrieve relevant context

Add a cosine function and a retrieve helper to the same file:

function cosine(a: number[], b: number[]): number {
  const dot = a.reduce((sum, v, i) => sum + v * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, v) => sum + v * v, 0));
  const magB = Math.sqrt(b.reduce((sum, v) => sum + v * v, 0));
  return dot / (magA * magB);
}

export async function retrieve(query: string, index: Chunk[], topK = 2) {
  const { embedding } = await embed({
    model: openai.embedding('text-embedding-3-small'),
    value: query,
  });
  return index
    .map((c) => ({ ...c, score: cosine(embedding, c.vector) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);
}

Stream chat with injected context

Create the route src/app/api/chat/route.ts. The Vercel AI SDK expects messages in the request body (the useChat hook sends this shape). We grab the last user message, retrieve context, and prepend a system prompt.

import { streamText, type Message } from 'ai';
import { openai } from '@ai-sdk/openai';
import { buildIndex, retrieve } from '@/lib/rag';

const index = await buildIndex();

export async function POST(req: Request) {
  const { messages }: { messages: Message[] } = await req.json();
  const lastUser = [...messages].reverse().find((m) => m.role === 'user');
  const context = lastUser ? await retrieve(lastUser.content, index) : [];

  const system = [
    'You answer questions using only the provided context.',
    'If the context is insufficient, say you do not know.',
    'Context:',
    ...context.map((c) => `- ${c.text}`),
  ].join('\n');

  const result = await streamText({
    model: openai('gpt-4o-mini'),
    system,
    messages,
  });

  return result.toDataStreamResponse();
}

toDataStreamResponse() emits the protocol that @ai-sdk/react’s useChat consumes out of the box.

Build the UI

Replace src/app/page.tsx with a client component:

'use client';

import { useChat } from '@ai-sdk/react';

export default function Page() {
  const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat();

  return (
    <main className="max-w-2xl mx-auto p-4">
      <h1 className="text-xl font-bold mb-4">RAG Chat</h1>
      <div className="space-y-2 mb-4">
        {messages.map((m) => (
          <div key={m.id} className="whitespace-pre-wrap">
            <span className="font-semibold">{m.role}: </span>
            {m.content}
          </div>
        ))}
        {isLoading && <div className="text-gray-500">streaming…</div>}
      </div>
      <form onSubmit={handleSubmit} className="flex gap-2">
        <input
          className="flex-1 border rounded px-2 py-1"
          value={input}
          onChange={handleInputChange}
          placeholder="Ask about the SDK"
        />
        <button type="submit" className="bg-black text-white px-3 rounded">
          Send
        </button>
      </form>
    </main>
  );
}

Run npm run dev and open http://localhost:3000. You have a functional next.js rag chatbot vercel ai sdk stack.

Checkpoint: retrieval output

Before trusting the chat, verify retrieval independently. Temporarily add a script:

// src/lib/check.ts
import { buildIndex, retrieve } from './rag';

async function main() {
  const idx = await buildIndex();
  const hits = await retrieve('How do I stream from an LLM?', idx);
  console.log(hits.map((h) => `${h.score.toFixed(3)}: ${h.text}`));
}
main();

Expected output (scores vary slightly by embedding version):

0.842: The Vercel AI SDK provides a unified API for streaming text from LLMs.
0.611: App Router server actions can securely call model providers without exposing keys.

If the top hit is irrelevant, tune chunk size or topK.

Checkpoint: chat streaming

Ask the UI: “What does cosine similarity ignore?”

The streamed answer should ground in the retrieved chunk:

user: What does cosine similarity ignore?
assistant: Cosine similarity measures the angle between two vectors and ignores their magnitude.

Because the system prompt forbids outside knowledge, the bot will refuse questions like “What’s the capital of France?” unless that fact is in your docs. That’s the RAG guardrail working.

Using a model gateway for production

The code above calls OpenAI directly. If you want automatic fallback when a provider is rate-limited, point the SDK at an OpenAI-compatible endpoint like n4n.ai, which fronts 240+ models and fails over without code changes. Swap the provider definition:

import { createOpenAI } from '@ai-sdk/openai';

const gateway = createOpenAI({
  baseURL: 'https://api.n4n.ai/v1',
  apiKey: process.env.N4N_API_KEY,
});

// then use gateway('gpt-4o-mini') and gateway.embedding('text-embedding-3-small')

The Vercel AI SDK sends the same request shape, so streamText and embed need no other modifications. You also get per-token metering from the gateway instead of rolling your own.

Caveats and next steps

• Chunking: real corpora need sentence-aware splitting. Use pdf-parse or unstructured before embedding.
• Latency: buildIndex() runs on first request. Move it to a scheduled job or a singleton outside the hot path.
• Re-ranking: cosine is a weak signal. Add a cross-encoder re-rank if precision drops.
• Cache: the SDK honors provider cache-control hints; pass cacheControl on embeddings if your gateway supports it.

The next.js rag chatbot vercel ai sdk pattern stays the same as you scale: retrieve synchronously in the route, inject, stream. Keep the index behind a fast lookup and the model behind a gateway, and the App Router handles the rest.