Streaming token-by-token responses with useChat

Streaming token-by-token responses transforms a chat interface from feeling like a form submission into a live conversation. This usechat token streaming tutorial walks through building a production-ready streaming chat UI with the Vercel AI SDK’s useChat hook, covering the backend streaming endpoint, the frontend hook configuration, and the details that keep the experience smooth under real load.

Step 1: Set up the project and dependencies

Start with a Next.js App Router project. The AI SDK works with Pages Router too, but App Router’s native streaming support makes the integration cleaner.

npx create-next-app@latest streaming-chat --typescript --tailwind --eslint --app --src-dir --import-alias "@/*"
cd streaming-chat
npm install ai @ai-sdk/openai

The ai package contains useChat and the streaming helpers. @ai-sdk/openai provides the OpenAI-compatible model interface — swap this for @ai-sdk/anthropic, @ai-sdk/google, or any other provider without changing your frontend code.

If you’re routing through a gateway like n4n.ai that exposes an OpenAI-compatible endpoint across 240+ models, you only need to change the base URL and API key in the model configuration. The streaming contract stays identical.

Step 2: Create the streaming API route

In src/app/api/chat/route.ts, build a POST handler that streams tokens back to the client. The AI SDK’s streamText function handles the provider communication and returns a ReadableStream compatible with the DataStreamResponse wrapper.

// src/app/api/chat/route.ts
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

export const maxDuration = 30;

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: openai('gpt-4o-mini'),
    messages,
    temperature: 0.7,
    maxTokens: 500,
  });

  return result.toDataStreamResponse();
}

Key points:

• maxDuration extends the Vercel function timeout to 30 seconds — streaming responses can exceed the default 10-second limit on hobby plans.
• streamText accepts the full message history, so context carries across turns automatically.
• toDataStreamResponse() returns a Response object with the proper text/event-stream content type and the SSE formatting that useChat expects.

If your provider requires custom headers (for example, routing directives or cache-control hints), pass them through the model settings:

const result = streamText({
  model: openai('gpt-4o-mini', {
    baseURL: 'https://api.n4n.ai/v1',
    headers: {
      'x-router-model': 'auto',
      'x-prefer-cache': 'true',
    },
  }),
  messages,
});

The frontend doesn’t change — only the model initialization shifts.

Step 3: Build the chat interface with useChat

Create src/app/page.tsx with a minimal chat component. The useChat hook manages message state, input handling, and the streaming connection lifecycle.

// src/app/page.tsx
'use client';

import { useChat } from 'ai/react';
import { useState } from 'react';

export default function Chat() {
  const { messages, input, handleInputChange, handleSubmit, isLoading, error } = useChat({
    api: '/api/chat',
    onError: (err) => {
      console.error('Chat error:', err);
      alert('Failed to send message. Check console for details.');
    },
  });

  return (
    <div className="flex flex-col h-screen bg-gray-50">
      <header className="p-4 bg-white border-b">
        <h1 className="text-xl font-semibold">Streaming Chat</h1>
      </header>

      <main className="flex-1 overflow-y-auto p-4 space-y-4">
        {messages.map((message) => (
          <div
            key={message.id}
            className={`flex ${message.role === 'assistant' ? 'justify-start' : 'justify-end'}`}
          >
            <div
              className={`max-w-[70%] rounded-2xl px-4 py-2 ${
                message.role === 'assistant'
                  ? 'bg-white text-gray-900 shadow-sm'
                  : 'bg-blue-600 text-white'
              }`}
            >
              {message.content}
            </div>
          </div>
        ))}

        {isLoading && (
          <div className="flex justify-start">
            <div className="bg-white rounded-2xl px-4 py-2 shadow-sm animate-pulse">
              <span className="text-gray-400">▌</span>
            </div>
          </div>
        )}

        {error && (
          <div className="text-center text-red-500 text-sm">
            Error: {error.message}
          </div>
        )}
      </main>

      <form onSubmit={handleSubmit} className="p-4 bg-white border-t">
        <div className="flex gap-2 max-w-3xl mx-auto">
          <input
            value={input}
            onChange={handleInputChange}
            placeholder="Type a message..."
            className="flex-1 px-4 py-2 border rounded-xl focus:outline-none focus:ring-2 focus:ring-blue-500"
            disabled={isLoading}
          />
          <button
            type="submit"
            disabled={isLoading || !input.trim()}
            className="px-6 py-2 bg-blue-600 text-white rounded-xl hover:bg-blue-700 disabled:opacity-50 disabled:cursor-not-allowed transition-colors"
          >
            Send
          </button>
        </div>
      </form>
    </div>
  );
}

What useChat handles for you:

• Message state: messages array stays in sync with the stream. Each assistant message arrives as a single object that updates incrementally — you don’t manually append chunks.
• Input management: input, handleInputChange, and handleSubmit wire the form without boilerplate.
• Loading state: isLoading reflects whether a stream is active. The example shows a typing indicator while tokens arrive.
• Error boundary: The onError callback catches network failures, provider errors, and parsing issues.

Run the dev server and verify the baseline works:

npm run dev

Open http://localhost:3000, send a message, and confirm tokens appear character-by-character rather than in a single block.

Step 4: Handle streaming interruptions and retries

Real networks drop connections. The useChat hook automatically retries failed requests once by default, but you can customize this behavior and expose retry controls to the user.

Update the hook configuration:

const { messages, input, handleInputChange, handleSubmit, isLoading, error, stop, reload } = useChat({
  api: '/api/chat',
  maxRetries: 3,
  retryDelay: (attempt) => Math.min(1000 * 2 ** attempt, 10000),
  onError: (err) => {
    console.error('Chat error:', err);
  },
});

• maxRetries: Number of automatic retry attempts before surfacing an error.
• retryDelay: Exponential backoff function. The example caps at 10 seconds.
• stop: Function to abort the current stream. Useful for a “Stop generating” button.
• reload: Function to re-send the last user message. Useful for a “Regenerate” button.

Add the controls to the UI:

{isLoading && (
  <div className="flex justify-start">
    <div className="bg-white rounded-2xl px-4 py-2 shadow-sm flex items-center gap-2">
      <span className="text-gray-400 animate-pulse">▌</span>
      <button
        onClick={stop}
        className="text-xs text-gray-500 hover:text-gray-700"
        aria-label="Stop generating"
      >
        Stop
      </button>
    </div>
  </div>
)}

{error && (
  <div className="flex justify-center gap-2">
    <p className="text-red-500 text-sm">{error.message}</p>
    <button
      onClick={reload}
      className="text-xs text-blue-600 hover:underline"
    >
      Retry
    </button>
  </div>
)}

Test interruption handling by throttling your network in DevTools (Network tab → “Slow 3G”) and sending a long request, then clicking Stop. The stream aborts cleanly and the partial message remains in history.

Step 5: Persist conversation history across sessions

By default, useChat keeps messages in React state — a refresh wipes the conversation. For a production chat, persist to localStorage or a backend database.

Option A: localStorage (client-only persistence)

'use client';

import { useChat } from 'ai/react';
import { useEffect, useState } from 'react';

export default function Chat() {
  const [hydrated, setHydrated] = useState(false);

  const { messages, setMessages, input, handleInputChange, handleSubmit, isLoading, error, stop, reload } = useChat({
    api: '/api/chat',
    initialMessages: [],
    onFinish: (message) => {
      // Persist after each completed turn
      localStorage.setItem('chat-messages', JSON.stringify(message));
    },
  });

  useEffect(() => {
    const stored = localStorage.getItem('chat-messages');
    if (stored) {
      try {
        setMessages(JSON.parse(stored));
      } catch {
        localStorage.removeItem('chat-messages');
      }
    }
    setHydrated(true);
  }, [setMessages]);

  if (!hydrated) {
    return <div className="flex h-screen items-center justify-center">Loading…</div>;
  }

  // ... render same as before
}

The onFinish callback fires when the stream completes successfully. setMessages replaces the entire history atomically, avoiding flicker during hydration.

Option B: Server-backed persistence

For multi-device sync, store conversations in a database. The pattern: create a conversation record on first message, append each turn, and load history on page mount.

// src/app/api/conversations/route.ts
import { db } from '@/lib/db'; // your ORM/client

export async function POST(req: Request) {
  const { title } = await req.json();
  const conversation = await db.conversation.create({
    data: { title: title ?? 'New chat' },
  });
  return Response.json(conversation);
}

export async function GET(req: Request) {
  const { searchParams } = new URL(req.url);
  const id = searchParams.get('id');
  if (!id) return new Response('Missing id', { status: 400 });

  const conversation = await db.conversation.findUnique({
    where: { id },
    include: { messages: { orderBy: { createdAt: 'asc' } } },
  });
  return Response.json(conversation);
}

Then hydrate useChat with initialMessages fetched from the conversation endpoint. The hook accepts messages in the same { id, role, content } format it produces.

Step 6: Add tool calls and structured outputs

Streaming isn’t limited to plain text. The AI SDK supports tool calls that stream arguments incrementally, letting the UI render partial tool invocations (e.g., a search query building character-by-character).

Extend the API route:

// src/app/api/chat/route.ts
import { streamText, tool } from 'ai';
import { openai } from '@ai-sdk/openai';
import { z } from 'zod';

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: openai('gpt-4o-mini'),
    messages,
    tools: {
      search: tool({
        parameters: z.object({
          query: z.string(),
        }),
        execute: async ({ query }) => {
          // Replace with real search API
          return { results: [`Result for: ${query}`] };
        },
      }),
    },
    maxSteps: 3, // Allow multi-step tool use
  });

  return result.toDataStreamResponse();
}

On the frontend, tool calls appear in message.parts with type 'tool-invocation'. Render them distinctly:

{messages.map((message) => (
  <div key={message.id} className="flex justify-start">
    <div className="max-w-[70%] bg-white rounded-2xl px-4 py-2 shadow-sm">
      {message.content}
      {message.parts?.map((part, i) => (
        part.type === 'tool-invocation' && (
          <details key={i} className="mt-2 text-sm border rounded p-2 bg-gray-50">
            <summary className="font-mono text-blue-600 cursor-pointer">
              Tool: {part.toolInvocation.toolName}
            </summary>
            <pre className="mt-1 whitespace-pre-wrap">
              {JSON.stringify(part.toolInvocation.args, null, 2)}
            </pre>
            {part.toolInvocation.state === 'result' && (
              <pre className="mt-1 whitespace-pre-wrap text-green-700">
                {JSON.stringify(part.toolInvocation.result, null, 2)}
              </pre>
            )}
          </details>
        )
      ))}
    </div>
  </div>
))}

The tool-invocation part streams through states: 'partial-call' → 'call' → 'result'. The UI updates live as arguments arrive and again when the tool returns.

Step 7: Optimize for perceived latency

Three techniques make streaming feel faster without changing the model:

1. Show a thinking indicator immediately

The isLoading flag turns true before the first token arrives. Render a skeleton or typing animation instantly:

{isLoading && messages.length === 0 && (
  <div className="flex justify-start">
    <div className="bg-white rounded-2xl px-4 py-2 shadow-sm flex items-center gap-1">
      <span className="w-2 h-2 bg-gray-300 rounded-full animate-bounce" style={{animationDelay: '0ms'}} />
      <span className="w-2 h-2 bg-gray-300 rounded-full animate-bounce" style={{animationDelay: '150ms'}} />
      <span className="w-2 h-2 bg-gray-300 rounded-full animate-bounce" style={{animationDelay: '300ms'}} />
    </div>
  </div>
)}

2. Preconnect to the API origin

In src/app/layout.tsx, add a preconnect hint for your API domain (especially important if the model runs on a different origin):

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <head>
        <link rel="preconnect" href="https://api.openai.com" crossOrigin="anonymous" />
        {/* Or your gateway: https://api.n4n.ai */}
      </head>
      <body>{children}</body>
    </html>
  );
}

3. Enable response compression

If you self-heavy streaming (long assistant messages), ensure your edge function or server sends compressed chunks. On Vercel, this is automatic. On self-hosted Node, add compression middleware:

// server.ts (if using custom server)
import compression from 'compression';
app.use(compression());

Step 8: Verify the implementation end to end

Run through this checklist in your browser:

1. Cold start: Open the page, send “Hello”. First token appears within 500ms on a warm edge function; cold start may take 1-2s.
2. Streaming continuity: Send a request that generates 200+ tokens. Verify no gaps or stalls — the cursor should move smoothly.
3. Interruption: Send a long request, click Stop mid-stream. The partial message remains. Click Retry — a new stream starts from the same user message.
4. Tool calls: Trigger a tool (e.g., “Search for today’s weather”). Watch the tool invocation panel populate arguments incrementally, then show the result.
5. Persistence: Refresh the page. Conversation history restores from localStorage (or your database).
6. Error handling: Disconnect network, send a message. Error toast appears. Reconnect, click Retry — message sends successfully.
7. Mobile layout: Test in device toolbar. Input stays accessible, messages wrap correctly, touch targets meet 44px minimum.

Step 9: Deploy and monitor

Deploy to Vercel (or your platform of choice). The API route runs as a serverless function with streaming enabled by default.

Monitor these signals in production:

• Time to first token (TTFT): Target < 800ms p95. High TTFT usually means cold starts or provider latency.
• Stream completion rate: Percentage of streams that finish without client-side error. Below 99% indicates network or provider issues.
• Retry rate: How often maxRetries is exhausted. Spikes suggest provider degradation — consider automatic fallback routing.
• Token throughput: Tokens/second per stream. Drops can signal provider throttling.

If you’re routing through a gateway that honors provider cache-control hints and supports automatic fallback, configure alerts on fallback activation rate — it’s an early warning that your primary provider is degrading.

This usechat token streaming tutorial covers the full stack: a streaming API route, a resilient frontend with useChat, persistence, tool calls, and production hardening. The patterns here scale from a demo to a high-traffic chat product without rewriting the streaming layer. Start with the minimal implementation in Steps 1-3, then layer on the reliability and UX improvements as your requirements grow.