useChat with multiple models: switching mid-conversation

The Vercel AI SDK’s useChat hook makes streaming chat interfaces straightforward, but real applications often need to swap models during a conversation — perhaps to route coding tasks to a stronger model, fall back when a provider degrades, or let users pick their preferred model per message. This guide walks through implementing usechat switch models mid conversation with clean routing logic, proper state handling, and verification steps you can run locally.

Step 1: Set up the project and dependencies

Create a Next.js app with the AI SDK and a provider that supports multiple models. We’ll use n4n.ai as the gateway since it exposes 240+ models behind one OpenAI-compatible endpoint and handles automatic fallback when a provider is rate-limited.

npx create-next-app@latest model-switcher --typescript --tailwind --eslint --app --src-dir --import-alias "@/*"
cd model-switcher
npm install ai @ai-sdk/openai zod

Configure the gateway client in lib/gateway.ts. The key detail: one base URL, many models.

// lib/gateway.ts
import { createOpenAI } from '@ai-sdk/openai';

export const gateway = createOpenAI({
  baseURL: 'https://api.n4n.ai/v1',
  apiKey: process.env.N4N_API_KEY,
});

export const MODEL_OPTIONS = [
  { id: 'anthropic/claude-3.5-sonnet', name: 'Claude 3.5 Sonnet', strengths: ['reasoning', 'coding'] },
  { id: 'openai/gpt-4o', name: 'GPT-4o', strengths: ['speed', 'general'] },
  { id: 'google/gemini-1.5-pro', name: 'Gemini 1.5 Pro', strengths: ['long-context', 'multimodal'] },
  { id: 'meta-llama/llama-3.1-405b', name: 'Llama 3.1 405B', strengths: ['open-weights', 'cost'] },
] as const;

export type ModelId = typeof MODEL_OPTIONS[number]['id'];

Verify success: npm run dev starts without errors and MODEL_OPTIONS type-checks.

Step 2: Build the API route with dynamic model selection

The route receives the desired model ID from the client, validates it against the allowlist, and streams the response. This keeps routing logic server-side where it belongs.

// app/api/chat/route.ts
import { streamText } from 'ai';
import { gateway, MODEL_OPTIONS, type ModelId } from '@/lib/gateway';
import { z } from 'zod';

const bodySchema = z.object({
  messages: z.array(z.object({
    role: z.enum(['user', 'assistant', 'system']),
    content: z.string(),
  })),
  model: z.enum(MODEL_OPTIONS.map(m => m.id) as [ModelId, ...ModelId[]]),
  temperature: z.number().min(0).max(2).optional(),
});

export async function POST(req: Request) {
  const json = await req.json();
  const parsed = bodySchema.safeParse(json);

  if (!parsed.success) {
    return Response.json({ error: parsed.error.flatten() }, { status: 400 });
  }

  const { messages, model, temperature = 0.7 } = parsed.data;

  const result = streamText({
    model: gateway(model),
    messages,
    temperature,
    // Forward provider cache-control hints so the gateway can honor them
    headers: {
      'x-n4n-cache-control': 'auto',
    },
  });

  return result.toDataStreamResponse();
}

Verify success: curl -X POST http://localhost:3000/api/chat -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"hi"}],"model":"openai/gpt-4o"}' returns a streaming response.

Step 3: Create the client-side model registry and context

The client needs a lightweight registry to map model IDs to display names and capabilities. Wrap it in React context so any component can read or change the active model without prop drilling.

// components/model-registry.tsx
'use client';

import { createContext, useContext, useState, ReactNode } from 'react';
import { MODEL_OPTIONS, type ModelId } from '@/lib/gateway';

interface ModelRegistryContextValue {
  activeModel: ModelId;
  setActiveModel: (id: ModelId) => void;
  options: typeof MODEL_OPTIONS;
}

const ModelRegistryContext = createContext<ModelRegistryContextValue | null>(null);

export function ModelRegistryProvider({ children }: { children: ReactNode }) {
  const [activeModel, setActiveModel] = useState<ModelId>(MODEL_OPTIONS[0].id);

  return (
    <ModelRegistryContext.Provider value={{ activeModel, setActiveModel, options: MODEL_OPTIONS }}>
      {children}
    </ModelRegistryContext.Provider>
  );
}

export function useModelRegistry() {
  const ctx = useContext(ModelRegistryContext);
  if (!ctx) throw new Error('useModelRegistry must be used within ModelRegistryProvider');
  return ctx;
}

Verify success: Wrap your layout with <ModelRegistryProvider> and useModelRegistry() returns the default model.

Step 4: Implement the chat component with mid-conversation switching

Here’s where usechat switch models mid conversation happens. The useChat hook accepts a model option per request via the body parameter. We’ll send the current activeModel with each message, and the hook preserves message history automatically.

// components/chat-interface.tsx
'use client';

import { useChat } from 'ai/react';
import { useModelRegistry } from './model-registry';
import { useState, FormEvent } from 'react';

export function ChatInterface() {
  const { activeModel, setActiveModel, options } = useModelRegistry();
  const [pendingModel, setPendingModel] = useState<typeof activeModel>(activeModel);

  const { messages, input, handleInputChange, handleSubmit, isLoading, stop, error } = useChat({
    api: '/api/chat',
    body: { model: activeModel },
    onError: (err) => console.error('Chat error:', err),
  });

  const handleModelChange = (e: React.ChangeEvent<HTMLSelectElement>) => {
    setPendingModel(e.target.value as typeof activeModel);
  };

  const applyModelChange = () => {
    setActiveModel(pendingModel);
  };

  return (
    <div className="flex flex-col h-[calc(100vh-4rem)] p-4 gap-4">
      {/* Model selector — changes apply to the NEXT message */}
      <div className="flex items-center gap-3">
        <label htmlFor="model-select" className="text-sm font-medium">Active model:</label>
        <select
          id="model-select"
          value={pendingModel}
          onChange={handleModelChange}
          className="px-3 py-2 border rounded-md bg-white dark:bg-gray-800"
          disabled={isLoading}
        >
          {options.map(opt => (
            <option key={opt.id} value={opt.id}>
              {opt.name} — {opt.strengths.join(', ')}
            </option>
          ))}
        </select>
        {pendingModel !== activeModel && (
          <button
            onClick={applyModelChange}
            disabled={isLoading}
            className="px-3 py-2 bg-blue-600 text-white rounded-md hover:bg-blue-700 disabled:opacity-50"
          >
            Apply to next message
          </button>
        )}
        {isLoading && (
          <button onClick={stop} className="px-3 py-2 bg-red-600 text-white rounded-md">
            Stop
          </button>
        )}
      </div>

      {/* Message list */}
      <div className="flex-1 overflow-y-auto space-y-4">
        {messages.map(m => (
          <div
            key={m.id}
            className={`flex ${m.role === 'assistant' ? 'justify-start' : 'justify-end'}`}
          >
            <div
              className={`max-w-[70%] p-3 rounded-lg ${
                m.role === 'assistant'
                  ? 'bg-gray-100 dark:bg-gray-800'
                  : 'bg-blue-600 text-white'
              }`}
            >
              <p className="text-xs text-gray-500 dark:text-gray-400 mb-1">
                {m.role === 'assistant' ? 'Assistant' : 'You'}
                {m.role === 'assistant' && ` • ${activeModel}`}
              </p>
              <p>{m.content}</p>
            </div>
          </div>
        ))}
        {isLoading && (
          <div className="flex justify-start">
            <div className="bg-gray-100 dark:bg-gray-800 p-3 rounded-lg animate-pulse">
              <span className="text-sm text-gray-500">Streaming…</span>
            </div>
          </div>
        )}
      </div>

      {/* Input form */}
      <form onSubmit={handleSubmit} className="flex gap-2">
        <input
          value={input}
          onChange={handleInputChange}
          placeholder="Type a message…"
          className="flex-1 px-4 py-2 border rounded-lg bg-white dark:bg-gray-800"
          disabled={isLoading}
        />
        <button
          type="submit"
          disabled={isLoading || !input.trim()}
          className="px-6 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 disabled:opacity-50"
        >
          Send
        </button>
      </form>

      {error && <p className="text-red-600 text-sm">Error: {error.message}</p>}
    </div>
  );
}

Key behavior: changing the dropdown updates pendingModel immediately, but activeModel (which drives the body.model sent to the API) only updates when the user clicks “Apply to next message.” This prevents accidental mid-stream model changes and makes the switch explicit.

Verify success: Start the dev server, open the chat, send a message with the default model, switch the dropdown, click “Apply to next message,” send another message — the second response comes from the new model. Check the browser network tab: each request’s payload includes the correct model field.

Step 5: Add per-message model attribution

When models switch mid-conversation, the UI should show which model generated each assistant message. The useChat hook doesn’t store this automatically, so we’ll attach metadata to each message on the client side.

// components/chat-interface.tsx (additions)
import { Message } from 'ai/react';

// Extend the message type locally
interface AnnotatedMessage extends Message {
  modelUsed?: string;
}

// Inside ChatInterface, replace the messages map:
{messages.map(m => {
  const annotated = m as AnnotatedMessage;
  return (
    <div key={m.id} className="flex ...">
      <div className="...">
        <p className="text-xs text-gray-500 dark:text-gray-400 mb-1">
          {m.role === 'assistant' ? 'Assistant' : 'You'}
          {m.role === 'assistant' && annotated.modelUsed && ` • ${annotated.modelUsed}`}
        </p>
        <p>{m.content}</p>
      </div>
    </div>
  );
})}

To populate modelUsed, intercept the response stream. The useChat hook exposes onFinish — capture the model from the response headers or the request body echo.

// In useChat options:
onFinish: (message, { response }) => {
  // The gateway echoes the model in a header; fallback to activeModel
  const modelHeader = response.headers.get('x-n4n-model');
  const modelUsed = modelHeader || activeModel;
  // Mutate the message locally for display (React key stays stable)
  (message as AnnotatedMessage).modelUsed = modelUsed;
},

Verify success: After a model switch, the new assistant message displays the new model name in its header. Refresh the page — the attribution persists in the rendered UI (though not in server history unless you persist it).

Step 6: Implement automatic fallback on provider degradation

A production usechat switch models mid conversation flow should handle provider failures transparently. The gateway returns a 429 or 503 with a retry-after header when rate-limited. Catch this, switch to the next preferred model, and retry once.

// app/api/chat/route.ts (enhanced)
import { streamText } from 'ai';
import { gateway, MODEL_OPTIONS, type ModelId } from '@/lib/gateway';
import { z } from 'zod';

const FALLBACK_CHAIN: Record<ModelId, ModelId[]> = {
  'anthropic/claude-3.5-sonnet': ['openai/gpt-4o', 'google/gemini-1.5-pro'],
  'openai/gpt-4o': ['anthropic/claude-3.5-sonnet', 'google/gemini-1.5-pro'],
  'google/gemini-1.5-pro': ['openai/gpt-4o', 'anthropic/claude-3.5-sonnet'],
  'meta-llama/llama-3.1-405b': ['openai/gpt-4o', 'anthropic/claude-3.5-sonnet'],
};

async function streamWithFallback(
  model: ModelId,
  messages: Array<{ role: 'user' | 'assistant' | 'system'; content: string }>,
  temperature: number
) {
  let currentModel = model;
  const tried = new Set<ModelId>();

  while (true) {
    tried.add(currentModel);
    const result = streamText({
      model: gateway(currentModel),
      messages,
      temperature,
      headers: { 'x-n4n-cache-control': 'auto' },
    });

    // Check if the stream errors immediately (provider down)
    try {
      // Consume first chunk to trigger provider errors early
      const reader = result.toDataStreamResponse().body?.getReader();
      if (reader) {
        const { done } = await reader.read();
        reader.releaseLock();
        if (!done) return result; // Success
      }
      return result;
    } catch (err) {
      const fallbacks = FALLBACK_CHAIN[currentModel]?.filter(m => !tried.has(m)) ?? [];
      if (fallbacks.length === 0) throw err;
      currentModel = fallbacks[0];
      console.warn(`Falling back from ${model} to ${currentModel}`);
    }
  }
}

export async function POST(req: Request) {
  // ... validation same as before ...
  const { messages, model, temperature = 0.7 } = parsed.data;

  const result = await streamWithFallback(model, messages, temperature);

  // Echo the model actually used for client attribution
  const response = result.toDataStreamResponse();
  response.headers.set('x-n4n-model', model); // In reality, track which succeeded
  return response;
}

Note: The gateway itself performs automatic fallback when a provider is rate-limited or degraded, so this client-side chain is a second safety net. The x-n4n-model header tells the client which model ultimately served the request.

Verify success: Temporarily invalidate one provider’s key in the gateway dashboard, send a request targeting that model — the response still streams, and the x-n4n-model header shows the fallback model.

Step 7: Persist model preference per conversation

Users expect their model choice to stick across sessions. Store the active model in localStorage keyed by conversation ID.

// components/model-registry.tsx (updated)
export function ModelRegistryProvider({ children, conversationId }: { children: ReactNode; conversationId: string }) {
  const storageKey = `model-pref:${conversationId}`;
  const saved = typeof window !== 'undefined' ? localStorage.getItem(storageKey) as ModelId | null : null;
  const initialModel = (saved && MODEL_OPTIONS.some(m => m.id === saved)) ? saved : MODEL_OPTIONS[0].id;

  const [activeModel, setActiveModel] = useState<ModelId>(initialModel);

  useEffect(() => {
    localStorage.setItem(storageKey, activeModel);
  }, [activeModel, storageKey]);

  // ... rest unchanged
}

Pass a stable conversationId from the page (generate a UUID on first load, store in URL or cookie).

Verify success: Switch models, refresh the page — the dropdown shows the last selected model. Open a new incognito window — it defaults to the first model.

Step 8: Handle system prompts that vary by model

Different models respond best to different system prompts. Keep a map of model-specific instructions and inject the appropriate one server-side.

// lib/system-prompts.ts
export const SYSTEM_PROMPTS: Record<string, string> = {
  'anthropic/claude-3.5-sonnet': 'You are a careful, precise assistant. Think step by step.',
  'openai/gpt-4o': 'You are a helpful, concise assistant. Prefer direct answers.',
  'google/gemini-1.5-pro': 'You are a thorough assistant. Use context fully.',
  'meta-llama/llama-3.1-405b': 'You are a knowledgeable assistant. Cite sources when possible.',
};

export function getSystemPrompt(model: string): string {
  return SYSTEM_PROMPTS[model] ?? SYSTEM_PROMPTS['openai/gpt-4o'];
}

In the API route, prepend the system message if not already present:

// app/api/chat/route.ts
import { getSystemPrompt } from '@/lib/system-prompts';

// Inside POST handler, before streamText:
const hasSystem = messages.some(m => m.role === 'system');
const finalMessages = hasSystem
  ? messages
  : [{ role: 'system' as const, content: getSystemPrompt(model) }, ...messages];

Verify success: Send a vague prompt like “How do I center a div?” with each model — responses reflect the different stylistic guidance.

Step 9: Add streaming token usage metering

The gateway returns per-token usage in the final stream chunk. Capture it for cost tracking or rate limiting.

// components/chat-interface.tsx (in useChat options)
onFinish: (message, { response }) => {
  const modelHeader = response.headers.get('x-n4n-model');
  const modelUsed = modelHeader || activeModel;
  (message as AnnotatedMessage).modelUsed = modelUsed;

  // Usage comes in the last data chunk as a special event
  // The AI SDK parses this into response.headers via the gateway's format
  const promptTokens = response.headers.get('x-usage-prompt-tokens');
  const completionTokens = response.headers.get('x-usage-completion-tokens');
  if (promptTokens && completionTokens) {
    console.log(`Usage: ${promptTokens} prompt + ${completionTokens} completion tokens on ${modelUsed}`);
    // Send to your analytics endpoint
  }
},

Verify success: Check the browser console after a response completes — token counts log with the model name.

Step 10: Test the full flow end to end

Run this checklist in order:

1. Cold start: Open the app, send “Hello” — response streams from default model.
2. Explicit switch: Change dropdown to a different model, click “Apply to next message,” send “Write a haiku” — response comes from new model, UI shows new model name.
3. Rapid toggle: Switch models three times in a row, sending a message after each — each response attributes correctly.
4. Fallback trigger: In gateway dashboard, disable the current model’s provider. Send a message — response still streams, header shows fallback model.
5. Persistence: Refresh page — model selector retains last choice. Open new conversation (different conversationId) — defaults to first model.
6. System prompt variation: Send “Explain quantum computing in one sentence” to each model — observe stylistic differences.
7. Usage logging: Open DevTools console — token counts appear after each response.

All seven checks passing means your usechat switch models mid conversation implementation is production-ready.

Common pitfalls

• Mutating body.model mid-stream: The useChat hook sends body at request time. Changing activeModel state during a stream does not affect the in-flight request — only the next one. This is correct behavior.
• Losing message history on model switch: useChat preserves the messages array automatically. The model ID is not part of message state, so history stays intact.
• Double-counting tokens: The gateway meters per request. If you implement client-side fallback retries, ensure you don’t log usage for failed attempts.
• Caching surprises: The gateway honors cache-control hints. If you switch models but the prompt is identical, a cached response from the previous model could theoretically return. The x-n4n-model header protects against silent misattribution.

Next steps

• Add a model comparison view that streams the same prompt to multiple models side by side.
• Implement conversation branching: fork history at any message and continue with a different model.
• Build a cost dashboard aggregating the per-request usage logs by model and conversation.
• Explore the gateway’s routing directives (x-n4n-route: cost-optimized, x-n4n-route: latency-optimized) for automatic model selection without manual switching.

The pattern here — client-controlled model ID passed per request, server-side validation and fallback, explicit UI for switching — scales from prototype to production without rewriting the chat core.