Testing a Vercel AI SDK chatbot end to end means verifying the full request path: user input hits your Next.js route, the SDK streams tokens from the model provider, tool calls execute, and the UI renders each chunk without flicker or data loss. Unit tests catch logic bugs; only an end-to-end suite catches the integration failures that happen when streaming meets network latency, provider fallbacks, or race conditions in the client store. This guide walks through a minimal, production-shaped test suite you can drop into a Next.js app using the AI SDK and Playwright.
Step 1: Scaffold the test infrastructure
Start with a fresh Next.js 14+ app using the App Router and the AI SDK installed. If you already have a project, skip to the dependency install.
npx create-next-app@latest ai-chatbot --typescript --tailwind --eslint --app --src-dir --import-alias "@/*"
cd ai-chatbot
npm i ai @ai-sdk/openai zod
npm i -D playwright @playwright/test @types/node
npx playwright install --with-deps chromiumCreate a playwright.config.ts at the repo root. Use the webServer option so Playwright boots your dev server before tests run and tears it down after.
// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: 'html',
use: {
baseURL: 'http://localhost:3000',
trace: 'on-first-retry',
},
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
],
webServer: {
command: 'npm run dev',
url: 'http://localhost:3000',
reuseExistingServer: !process.env.CI,
timeout: 120_000,
},
});Add a test script to package.json:
"scripts": {
"test:e2e": "playwright test",
"test:e2e:ui": "playwright test --ui"
}Step 2: Build a minimal chat route and page
The AI SDK’s streamText returns a ReadableStream that the client consumes via useChat. Keep the route tiny so the test surface stays clear.
// src/app/api/chat/route.ts
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { z } from 'zod';
export const maxDuration = 30;
export async function POST(req: Request) {
const { messages } = await req.json();
const result = await streamText({
model: openai('gpt-4o-mini'),
messages,
tools: {
getWeather: {
parameters: z.object({
location: z.string().describe('City name'),
unit: z.enum(['celsius', 'fahrenheit']).default('celsius'),
}),
execute: async ({ location, unit }) => {
// Deterministic stub for testing
const temp = unit === 'celsius' ? 22 : 72;
return { location, temperature: temp, unit, condition: 'sunny' };
},
},
},
});
return result.toDataStreamResponse();
}The page component uses useChat from @ai-sdk/react. Keep it in a client component.
// src/app/page.tsx
'use client';
import { useChat } from '@ai-sdk/react';
import { useState, FormEvent } from 'react';
export default function ChatPage() {
const { messages, input, handleInputChange, handleSubmit, isLoading, error } = useChat({
api: '/api/chat',
});
const [submitted, setSubmitted] = useState(false);
const onSubmit = (e: FormEvent<HTMLFormElement>) => {
e.preventDefault();
setSubmitted(true);
handleSubmit(e);
};
return (
<main className="flex min-h-screen flex-col items-center p-4">
<h1 className="mb-4 text-2xl font-semibold">Test Chatbot</h1>
<div className="w-full max-w-2xl space-y-3" data-testid="messages">
{messages.map((m) => (
<div key={m.id} className="p-3 rounded bg-gray-100" data-testid={`msg-${m.role}`}>
<strong className="capitalize">{m.role}:</strong>{' '}
<span data-testid={`msg-content-${m.role}`}>{m.content}</span>
{m.toolInvocations?.length && (
<div data-testid="tool-invocations">
{m.toolInvocations.map((t, i) => (
<div key={i} data-testid="tool-call">
<code>{t.toolName}</code>: {JSON.stringify(t.args)}
</div>
))}
</div>
)}
</div>
))}
</div>
<form onSubmit={onSubmit} className="w-full max-w-2xl flex gap-2" data-testid="chat-form">
<input
value={input}
onChange={handleInputChange}
placeholder="Type a message..."
className="flex-1 rounded border p-2"
data-testid="chat-input"
disabled={isLoading}
/>
<button type="submit" disabled={isLoading || !input.trim()} data-testid="send-btn">
Send
</button>
</form>
{error && <div className="text-red-600" data-testid="error">{error.message}</div>}
{submitted && !isLoading && <div data-testid="turn-complete" />}
</main>
);
}The data-testid attributes give Playwright stable selectors. The turn-complete marker appears after each assistant turn finishes, which the test will wait for.
Step 3: Write the happy-path streaming test
Create e2e/chat.spec.ts. The first test verifies that a user message streams back token-by-token and the final message lands in the DOM.
// e2e/chat.spec.ts
import { test, expect } from '@playwright/test';
test.describe('Chat streaming', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/');
await expect(page.getByTestId('chat-input')).toBeVisible();
});
test('streams assistant response token by token', async ({ page }) => {
// Send a simple prompt that triggers a short, deterministic reply
await page.getByTestId('chat-input').fill('Reply with exactly: pong');
await page.getByTestId('send-btn').click();
// Wait for the assistant message to appear and start streaming
const assistantMsg = page.getByTestId('msg-assistant').first();
await expect(assistantMsg).toBeVisible({ timeout: 10_000 });
// Poll until streaming finishes (turn-complete marker appears)
await expect(page.getByTestId('turn-complete')).toBeVisible({ timeout: 30_000 });
// Verify final content
const content = assistantMsg.getByTestId('msg-content-assistant');
await expect(content).toContainText('pong');
});
});Run it:
npm run test:e2eYou should see one passing test. The turn-complete div is the synchronization point — without it, the test would race the stream and flake.
Step 4: Test tool calling end to end
Tool calls are where integration breaks: the model emits a tool call chunk, the SDK executes the function, the result streams back as a tool result chunk, and the model emits the final answer. Verify the whole chain.
// e2e/chat.spec.ts (append)
test.describe('Tool calling', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/');
});
test('executes tool and streams final answer', async ({ page }) => {
await page.getByTestId('chat-input').fill('What is the weather in Tokyo?');
await page.getByTestId('send-btn').click();
// Tool call appears in the assistant message
const toolCall = page.getByTestId('tool-call').first();
await expect(toolCall).toBeVisible({ timeout: 15_000 });
await expect(toolCall).toContainText('getWeather');
await expect(toolCall).toContainText('Tokyo');
// Wait for the final answer after tool execution
await expect(page.getByTestId('turn-complete')).toBeVisible({ timeout: 30_000 });
const content = page.getByTestId('msg-content-assistant').first();
await expect(content).toContainText('Tokyo');
await expect(content).toContainText('sunny');
});
});The stub in the route returns deterministic data, so the assertion on “sunny” and “22”/“72” is safe. In a real suite you would swap the stub for a controlled test double via dependency injection or an environment flag.
Step 5: Verify error handling and recovery
Network blips, provider 5xx, and rate limits should surface as a user-visible error without crashing the stream consumer. The AI SDK throws an AIError that useChat catches into its error state.
// e2e/chat.spec.ts (append)
test.describe('Error handling', () => {
test('shows error when provider fails', async ({ page }) => {
await page.goto('/');
// Force a failure by hitting a non-existent model via a test-only route
// (You would add a test route that calls streamText with a bad model ID)
await page.route('/api/chat', async (route) => {
await route.fulfill({
status: 500,
contentType: 'application/json',
body: JSON.stringify({ error: 'Model unavailable' }),
});
});
await page.getByTestId('chat-input').fill('This will fail');
await page.getByTestId('send-btn').click();
await expect(page.getByTestId('error')).toBeVisible({ timeout: 5_000 });
await expect(page.getByTestId('error')).toContainText('Model unavailable');
});
});If you use a gateway that automatically falls back across providers (for example, n4n.ai honors client routing directives and forwards provider cache-control hints), you would test that the fallback path still streams correctly by forcing the primary provider to degrade and asserting the stream completes.
Step 6: Test multi-turn conversation context
The SDK sends the full message history on each request. Verify that the model sees prior turns.
// e2e/chat.spec.ts (append)
test.describe('Multi-turn context', () => {
test('remembers previous messages', async ({ page }) => {
await page.goto('/');
// Turn 1
await page.getByTestId('chat-input').fill('My name is Ada.');
await page.getByTestId('send-btn').click();
await expect(page.getByTestId('turn-complete')).toBeVisible({ timeout: 30_000 });
// Turn 2
await page.getByTestId('chat-input').fill('What is my name?');
await page.getByTestId('send-btn').click();
await expect(page.getByTestId('turn-complete')).toBeVisible({ timeout: 30_000 });
const content = page.getByTestId('msg-content-assistant').last();
await expect(content).toContainText('Ada');
});
});This test catches regressions where the history array is truncated, mutated, or not serialized correctly.
Step 7: Run in CI with GitHub Actions
Create .github/workflows/e2e.yml. The workflow installs dependencies, builds the Next.js app, and runs Playwright against the production build (not dev) to catch build-time errors.
# .github/workflows/e2e.yml
name: E2E Tests
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
timeout-minutes: 30
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: 'npm'
- run: npm ci
- run: npm run build
- run: npx playwright install --with-deps chromium
- run: npm run test:e2e
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- uses: actions/upload-artifact@v4
if: failure()
with:
name: playwright-report
path: playwright-report/
retention-days: 7Set OPENAI_API_KEY in the repository secrets. The test suite now runs on every PR.
Step 8: Flakiness guards and debugging
Flaky streaming tests usually stem from three sources: timing, selector brittleness, and shared state. Mitigate each:
- Timing — Always wait for a deterministic DOM signal (
turn-complete,tool-call,error) rather than arbitrary timeouts. Theturn-completepattern works because the client renders it afteruseChatsetsisLoadingto false. - Selectors — Use
data-testideverywhere. Never rely on CSS classes or text content that changes with design tweaks. - Shared state — Each test gets a fresh browser context by default. If you log in or set cookies in a
beforeAll, clean up inafterAllor usetest.isolate().
For local debugging, run the UI mode:
npm run test:e2e:uiPlaywright opens a trace viewer. Click a failed step to see the DOM snapshot, network waterfall, and console logs at that exact moment. The trace shows whether the stream chunks arrived, whether the tool call executed, and what the model actually returned.
Step 9: Extend with visual regression (optional)
Streaming UIs have subtle layout shifts: skeletons, token-by-token growth, markdown rendering mid-stream. Add @playwright/test visual snapshots to catch regressions.
// e2e/visual.spec.ts
import { test, expect } from '@playwright/test';
test('chat stream visual regression', async ({ page }) => {
await page.goto('/');
await page.getByTestId('chat-input').fill('Count to three.');
await page.getByTestId('send-btn').click();
await expect(page.getByTestId('turn-complete')).toBeVisible({ timeout: 30_000 });
// Snapshot the final rendered conversation
await expect(page.getByTestId('messages')).toHaveScreenshot('chat-final.png', {
maxDiffPixels: 100,
});
});Run once with --update-snapshots to establish baselines, then commit the PNGs. CI will fail if Tailwind changes or a markdown parser update shifts layout.
Step 10: Verify success locally and in CI
Checklist before merging:
-
npm run test:e2epasses locally againstdevserver. -
npm run build && npm run test:e2epasses against production build (CI does this). - No flakes on three consecutive CI runs.
- Trace artifacts upload on failure and are readable.
- Visual snapshots match (if enabled).
If all green, the suite is ready to catch the real integration bugs: stream corruption, tool-call serialization errors, history truncation, and provider fallback behavior.
You now have a minimal, maintainable end-to-end test suite for a Vercel AI SDK chatbot. The patterns — deterministic synchronization markers, data-testid selectors, Playwright webServer, and CI against the production build — scale to larger apps with authentication, RAG pipelines, and multi-model routing. Add tests as you add features; delete tests when you delete code. The suite stays fast because each test exercises a single user journey through the streaming pipeline.