Voice input transforms a chatbot from a typing exercise into something you can use while cooking, walking, or debugging with your hands full. The Vercel AI SDK handles the streaming LLM response side elegantly, but it doesn’t include speech recognition — you bring your own. This guide wires the browser’s Web Speech API into a Next.js App Router chatbot so users can tap a button, speak, and watch the model reply in real time.
Step 1: Scaffold the project and install dependencies
Start with a fresh Next.js 14+ project using the App Router and TypeScript. The AI SDK packages you need are ai for the streaming hooks and @ai-sdk/openai (or your provider of choice) for the model client.
npx create-next-app@latest voice-chatbot --typescript --tailwind --eslint --app --src-dir --import-alias "@/*"
cd voice-chatbot
npm install ai @ai-sdk/openai zodIf you prefer a different provider, swap @ai-sdk/openai for @ai-sdk/anthropic, @ai-sdk/google, or any AI SDK-compatible adapter. The streaming hook signatures stay the same.
Step 2: Create the streaming route handler
The AI SDK expects a POST endpoint that returns a ReadableStream of the model’s response. In the App Router, this lives under src/app/api/chat/route.ts.
// src/app/api/chat/route.ts
import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';
import { z } from 'zod';
export const maxDuration = 30;
const requestSchema = z.object({
messages: z.array(
z.object({
role: z.enum(['user', 'assistant', 'system']),
content: z.string(),
})
),
});
export async function POST(req: Request) {
const body = await req.json();
const parsed = requestSchema.safeParse(body);
if (!parsed.success) {
return Response.json({ error: 'Invalid request body' }, { status: 400 });
}
const { messages } = parsed.data;
const result = await streamText({
model: openai('gpt-4o-mini'),
messages,
temperature: 0.7,
maxTokens: 500,
});
return result.toDataStreamResponse();
}The toDataStreamResponse() helper emits the AI SDK’s wire protocol: text deltas, tool calls, and finish reasons as a single stream the client hook can consume. Keep the route lean — validation, model config, and streaming. Business logic belongs elsewhere.
Step 3: Build the voice recognition hook
The Web Speech API lives on window.SpeechRecognition (or webkitSpeechRecognition in Safari). Wrap it in a reusable hook so the component stays clean and you can test the recognition logic in isolation.
// src/hooks/useSpeechRecognition.ts
'use client';
import { useCallback, useRef, useState } from 'react';
type SpeechRecognitionStatus = 'idle' | 'listening' | 'processing' | 'error';
interface UseSpeechRecognitionOptions {
lang?: string;
continuous?: boolean;
interimResults?: boolean;
onResult?: (transcript: string, isFinal: boolean) => void;
onError?: (error: SpeechRecognitionErrorEvent) => void;
onEnd?: () => void;
}
export function useSpeechRecognition(options: UseSpeechRecognitionOptions = {}) {
const {
lang = 'en-US',
continuous = false,
interimResults = true,
onResult,
onError,
onEnd,
} = options;
const [status, setStatus] = useState<SpeechRecognitionStatus>('idle');
const [transcript, setTranscript] = useState('');
const recognitionRef = useRef<SpeechRecognition | null>(null);
const isListeningRef = useRef(false);
const start = useCallback(() => {
if (typeof window === 'undefined') return;
const SpeechRecognitionCtor =
window.SpeechRecognition || window.webkitSpeechRecognition;
if (!SpeechRecognitionCtor) {
setStatus('error');
onError?.(new Event('not-supported') as SpeechRecognitionErrorEvent);
return;
}
const recognition = new SpeechRecognitionCtor();
recognition.lang = lang;
recognition.continuous = continuous;
recognition.interimResults = interimResults;
recognition.onstart = () => {
isListeningRef.current = true;
setStatus('listening');
setTranscript('');
};
recognition.onresult = (event: SpeechRecognitionEvent) => {
let finalTranscript = '';
let interimTranscript = '';
for (let i = event.resultIndex; i < event.results.length; i++) {
const result = event.results[i];
if (result.isFinal) {
finalTranscript += result[0].transcript;
} else {
interimTranscript += result[0].transcript;
}
}
const combined = finalTranscript || interimTranscript;
setTranscript(combined);
onResult?.(combined, !!finalTranscript);
};
recognition.onerror = (event: SpeechRecognitionErrorEvent) => {
setStatus('error');
onError?.(event);
};
recognition.onend = () => {
isListeningRef.current = false;
setStatus('idle');
onEnd?.();
};
recognitionRef.current = recognition;
recognition.start();
}, [lang, continuous, interimResults, onResult, onError, onEnd]);
const stop = useCallback(() => {
recognitionRef.current?.stop();
}, []);
const abort = useCallback(() => {
recognitionRef.current?.abort();
isListeningRef.current = false;
setStatus('idle');
}, []);
return { status, transcript, start, stop, abort };
}This hook handles the browser prefix dance, surfaces interim results for live feedback, and cleanly tears down on unmount. The status state drives your button UI — idle, listening, processing, error.
Step 4: Wire the chat interface with useChat
The AI SDK’s useChat hook manages message history, streaming state, and the request lifecycle. Combine it with the speech hook in a single client component.
// src/components/ChatInterface.tsx
'use client';
import { useChat } from 'ai/react';
import { useSpeechRecognition } from '@/hooks/useSpeechRecognition';
import { useState, useRef, useEffect } from 'react';
export function ChatInterface() {
const { messages, input, handleInputChange, handleSubmit, isLoading, stop } =
useChat({
api: '/api/chat',
maxSteps: 5,
});
const [micPermission, setMicPermission] = useState<'prompt' | 'granted' | 'denied'>('prompt');
const transcriptRef = useRef('');
const voicesRef = useRef<SpeechSynthesisVoice[]>([]);
const { status, transcript, start, stop: stopListening, abort } =
useSpeechRecognition({
lang: 'en-US',
continuous: false,
interimResults: true,
onResult: (text, isFinal) => {
transcriptRef.current = text;
if (isFinal) {
handleInputChange({ target: { value: text } });
}
},
onError: (event) => {
console.error('Speech recognition error:', event.error);
if (event.error === 'not-allowed') {
setMicPermission('denied');
}
},
onEnd: () => {
if (transcriptRef.current.trim() && status !== 'error') {
handleSubmit(new Event('submit') as React.FormEvent<HTMLFormElement>);
}
},
});
useEffect(() => {
if (typeof window !== 'undefined' && 'speechSynthesis' in window) {
const loadVoices = () => {
voicesRef.current = window.speechSynthesis.getVoices();
};
loadVoices();
window.speechSynthesis.onvoiceschanged = loadVoices;
}
}, []);
const speak = useCallback((text: string) => {
if (typeof window === 'undefined' || !('speechSynthesis' in window)) return;
window.speechSynthesis.cancel();
const utterance = new SpeechSynthesisUtterance(text);
utterance.lang = 'en-US';
const preferredVoice = voicesRef.current.find(
(v) => v.lang.startsWith('en') && v.name.includes('Google')
);
if (preferredVoice) utterance.voice = preferredVoice;
window.speechSynthesis.speak(utterance);
}, []);
useEffect(() => {
const lastMessage = messages[messages.length - 1];
if (lastMessage?.role === 'assistant' && !isLoading) {
speak(lastMessage.content);
}
}, [messages, isLoading, speak]);
const handleVoiceClick = () => {
if (status === 'listening') {
stopListening();
} else if (status === 'idle') {
if (micPermission === 'denied') {
alert('Microphone access denied. Enable it in browser settings.');
return;
}
start();
}
};
const isListening = status === 'listening';
return (
<div className="flex flex-col h-[600px] w-full max-w-2xl mx-auto p-4 border rounded-xl bg-white dark:bg-gray-900">
<div className="flex-1 overflow-y-auto space-y-4 p-2">
{messages.map((message) => (
<div
key={message.id}
className={`flex ${message.role === 'assistant' ? 'justify-start' : 'justify-end'}`}
>
<div
className={`max-w-[80%] px-4 py-2 rounded-2xl ${
message.role === 'assistant'
? 'bg-gray-100 dark:bg-gray-800 text-gray-900 dark:text-gray-100 rounded-bl-none'
: 'bg-blue-600 text-white rounded-br-none'
}`}
>
<p className="whitespace-pre-wrap">{message.content}</p>
</div>
</div>
))}
{isLoading && (
<div className="flex justify-start">
<div className="bg-gray-100 dark:bg-gray-800 px-4 py-2 rounded-2xl rounded-bl-none animate-pulse">
<span className="text-gray-500 dark:text-gray-400">Thinking…</span>
</div>
</div>
)}
</div>
<form onSubmit={handleSubmit} className="flex gap-2 mt-4">
<input
value={input}
onChange={handleInputChange}
placeholder="Type or press the mic…"
className="flex-1 px-4 py-2 border rounded-full focus:outline-none focus:ring-2 focus:ring-blue-500"
disabled={isLoading || isListening}
/>
<button
type="button"
onClick={handleVoiceClick}
disabled={isLoading}
className={`p-3 rounded-full transition-colors ${
isListening
? 'bg-red-500 text-white animate-pulse'
: 'bg-gray-100 dark:bg-gray-800 hover:bg-gray-200 dark:hover:bg-gray-700'
}`}
aria-label={isListening ? 'Stop listening' : 'Start voice input'}
title={isListening ? 'Stop listening' : 'Start voice input'}
>
<svg
className="w-6 h-6"
fill="currentColor"
viewBox="0 0 24 24"
aria-hidden="true"
>
<path d="M12 14c1.66 0 3-1.34 3-3V5c0-1.66-1.34-3-3-3S9 3.34 9 5v6c0 1.66 1.34 3 3 3zm5.3-3c0 3-2.54 5.1-5.3 5.5V17c0 .55-.45 1-1 1s-1-.45-1-1v-1.5c-2.76-.4-5.3-2.5-5.3-5.5 0-2.64 2.05-4.78 4.6-4.97V5c0-.83.67-1.5 1.5-1.5s1.5.67 1.5 1.5v2.53c2.55.19 4.6 2.33 4.6 4.97z" />
</svg>
</button>
<button
type="submit"
disabled={!input.trim() || isLoading || isListening}
className="px-6 py-2 bg-blue-600 text-white rounded-full hover:bg-blue-700 disabled:opacity-50 disabled:cursor-not-allowed"
>
Send
</button>
</form>
{isListening && (
<div className="mt-2 text-center text-sm text-gray-600 dark:text-gray-400 font-mono">
Listening… <span className="text-red-500">{transcript || '—'}</span>
</div>
)}
{micPermission === 'denied' && (
<div className="mt-2 text-center text-sm text-red-500">
Microphone blocked. Click the lock icon in the address bar to allow access.
</div>
)}
</div>
);
}A few things worth noting:
- The
useChathook posts to/api/chatautomatically and streams deltas intomessages. - Voice input appends interim transcripts to the input field, then submits on final result via
onEnd. - The assistant’s final reply is spoken back using the Speech Synthesis API — optional but nice for a fully voice-driven loop.
- The mic button reflects
statusfrom the speech hook so users see listening state immediately.
Step 5: Add the page and layout
Drop the component into a page. The App Router requires a client boundary at the component level since both hooks use browser APIs.
// src/app/page.tsx
import { ChatInterface } from '@/components/ChatInterface';
export default function Home() {
return (
<main className="min-h-screen bg-gray-50 dark:bg-gray-950 flex items-center justify-center p-4">
<ChatInterface />
</main>
);
}// src/app/layout.tsx
import type { Metadata } from 'next';
import './globals.css';
export const metadata: Metadata = {
title: 'Voice Chatbot',
};
export default function RootLayout({
children,
}: {
children: React.ReactNode;
}) {
return (
<html lang="en" className="dark">
<body className="antialiased">{children}</body>
</html>
);
}Tailwind’s dark mode class strategy works well here — the component respects dark: variants automatically.
Step 6: Run and verify
Start the dev server:
npm run devOpen http://localhost:3000 in Chrome, Edge, or Safari. Click the microphone permission prompt — allow it. The verification checklist:
- Text path works — type a message, press Send, see streaming tokens appear.
- Voice path works — click the mic button, speak a short sentence (“What’s the weather in Tokyo?”), watch the interim transcript update live, then see the final transcript populate the input and auto-submit.
- Streaming response — the assistant’s reply streams token by token, then speaks aloud if you kept the
speak()effect. - Error handling — deny microphone permission, click mic, confirm the denied-state message appears.
- Abort behavior — click mic while listening; it stops and does not submit partial garbage.
If the mic button never enters the listening state, check the console for not-supported — the Web Speech API is unavailable in Firefox and some embedded webviews. Chrome and Safari cover the vast majority of desktop and mobile users.
Step 7: Harden for production
The demo works locally. Before shipping, address these gaps:
HTTPS required — The Web Speech API only works on localhost or secure origins. Deploy to Vercel (or any HTTPS host) and the permission prompt will appear.
VAD and silence detection — The native API’s continuous: false stops after a single utterance. For hands-free conversation, set continuous: true and implement your own voice activity detection: track interim result timestamps, stop after N seconds of silence, then submit. This avoids the “I stopped talking but it’s still listening” problem.
Provider fallback — If your primary model provider hits rate limits or latency spikes, the request fails. At n4n.ai we route through a gateway that automatically fails over to a healthy provider while preserving the same OpenAI-compatible stream format — the client code doesn’t change. You can implement a simpler version by catching the stream error and retrying with a secondary model.
Token metering — The AI SDK’s streamText returns usage on finish. Log it per conversation for cost tracking.
// Inside the route handler, after streamText:
const result = await streamText({ /* ... */ });
result.consumeStream(); // drain to get usage
const { usage } = await result.usage;
console.log({ promptTokens: usage.promptTokens, completionTokens: usage.completionTokens });Accessibility — The component includes aria-label on the mic button and a live region for the transcript. Add role="status" to the listening indicator so screen readers announce it.
Step 8: Extend with tool calling (optional)
Voice input shines when the model can act — check calendar, query a database, trigger a deploy. The AI SDK’s tools parameter in streamText lets you define functions the model can call. The streaming protocol emits tool call deltas, your client executes them, and the results feed back into the stream automatically.
// In the route handler
import { tool } from 'ai';
import { z } from 'zod';
const result = await streamText({
model: openai('gpt-4o-mini'),
messages,
tools: {
getWeather: tool({
parameters: z.object({ location: z.string() }),
execute: async ({ location }) => {
// Call your weather API
return { temperature: 72, condition: 'Sunny' };
},
}),
},
});The client needs no changes — useChat handles the multi-step loop. The user says “What’s the weather in Denver?”, the model calls getWeather, your function runs, and the final answer streams back. Voice becomes a genuine interface, not just a dictation layer.
You now have a production-ready voice chatbot: Next.js 14, Vercel AI SDK streaming, Web Speech API recognition, speech synthesis replies, and a clear path to tool calling and provider fallback. The pattern scales — swap the model, add RAG, plug in a gateway — without rewiring the voice layer.