Multi-model chat app: Claude, GPT-4o, and Mixtral 8x7B

You’re building a chat interface and your product team wants users to pick between Claude 3.5 Sonnet, GPT-4o, and Mixtral 8x7B on the fly. The Vercel AI SDK makes this straightforward, but the documentation scatters the pieces across multiple guides. This tutorial assembles them into a working Next.js app with model switching, streaming responses, tool calling, and graceful provider fallbacks — all in about 150 lines of code.

Prerequisites

• Node.js 18+ and pnpm (or npm/yarn)
• API keys for at least two providers: Anthropic, OpenAI, and/or a Mixtral host (Together AI, Fireworks, or self-hosted)
• Basic familiarity with Next.js App Router and React Server Components

Create the project and install dependencies:

pnpm create next-app@latest multi-model-chat --typescript --tailwind --eslint --app --src-dir --import-alias "@/*"
cd multi-model-chat
pnpm add ai @ai-sdk/anthropic @ai-sdk/openai @ai-sdk/mistral zod

The ai package is the Vercel AI SDK core. Provider packages (@ai-sdk/anthropic, @ai-sdk/openai, @ai-sdk/mistral) expose model factories that conform to the SDK’s LanguageModel interface. zod validates tool parameters.

Project structure

src/
├── app/
│   ├── api/
│   │   └── chat/
│   │       └── route.ts          # Streaming endpoint
│   ├── page.tsx                  # Client chat UI
│   └── layout.tsx
├── components/
│   ├── ModelSelector.tsx         # Dropdown to switch models
│   └── ChatInterface.tsx         # Message list + input
└── lib/
    ├── models.ts                 # Model registry + helpers
    └── tools.ts                  # Tool definitions

Define the model registry

Centralize model configuration so the UI and API route share a single source of truth. Each entry maps a friendly ID to a provider factory call.

// src/lib/models.ts
import { anthropic } from '@ai-sdk/anthropic';
import { openai } from '@ai-sdk/openai';
import { mistral } from '@ai-sdk/mistral';
import { LanguageModel } from 'ai';

export type ModelId = 'claude-3-5-sonnet' | 'gpt-4o' | 'mixtral-8x7b';

export const models: Record<ModelId, LanguageModel> = {
  'claude-3-5-sonnet': anthropic('claude-3-5-sonnet-20241022'),
  'gpt-4o': openai('gpt-4o'),
  'mixtral-8x7b': mistral('mistral-large-latest'), // or 'mixtral-8x7b-instruct' on Together/Fireworks
};

export const modelLabels: Record<ModelId, string> = {
  'claude-3-5-sonnet': 'Claude 3.5 Sonnet',
  'gpt-4o': 'GPT-4o',
  'mixtral-8x7b': 'Mixtral 8x7B',
};

export function getModel(id: string): LanguageModel {
  return models[id as ModelId] ?? models['gpt-4o'];
}

Checkpoint: Run pnpm build — it should compile cleanly. The registry exports a getModel helper that defaults to GPT-4o if an unknown ID arrives from the client.

Define a tool for demonstration

Tool calling works across all three providers when you use the SDK’s unified tools parameter. Here’s a weather lookup tool that returns structured data.

// src/lib/tools.ts
import { tool } from 'ai';
import { z } from 'zod';

export const weatherTool = tool({
  parameters: z.object({
    location: z.string().describe('City and state, e.g. "San Francisco, CA"'),
    unit: z.enum(['celsius', 'fahrenheit']).default('fahrenheit'),
  }),
  execute: async ({ location, unit }) => {
    // In production, call a real weather API. This mock keeps the tutorial self-contained.
    const tempF = Math.floor(Math.random() * 40) + 50;
    const tempC = Math.round((tempF - 32) * 5 / 9);
    return {
      location,
      temperature: unit === 'celsius' ? tempC : tempF,
      unit,
      condition: ['Sunny', 'Cloudy', 'Rainy', 'Partly cloudy'][Math.floor(Math.random() * 4)],
    };
  },
});

export const tools = { weather: weatherTool };

Build the streaming API route

The App Router route handler receives the selected model ID and message history, then streams the response using streamText. This is where provider fallbacks would live — see the note at the end.

// src/app/api/chat/route.ts
import { streamText } from 'ai';
import { getModel } from '@/lib/models';
import { tools } from '@/lib/tools';

export const maxDuration = 30;

export async function POST(req: Request) {
  const { messages, modelId } = await req.json();

  const model = getModel(modelId);

  const result = streamText({
    model,
    messages,
    tools,
    // Optional: system prompt per model
    system: modelId === 'claude-3-5-sonnet'
      ? 'You are a helpful assistant. Use tools when appropriate.'
      : undefined,
    // Optional: max tokens per provider
    maxTokens: modelId === 'mixtral-8x7b' ? 4096 : undefined,
  });

  return result.toDataStreamResponse();
}

Checkpoint: Start the dev server (pnpm dev) and POST to /api/chat with a test payload:

curl -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"What is the weather in Tokyo?"}],"modelId":"gpt-4o"}'

You should see a streaming SSE response with tool calls and text chunks.

Build the model selector component

A simple dropdown that writes the selection to localStorage and notifies the chat interface via a callback.

// src/components/ModelSelector.tsx
'use client';

import { ModelId, modelLabels } from '@/lib/models';
import { useState, useEffect } from 'react';

interface ModelSelectorProps {
  value: ModelId;
  onChange: (id: ModelId) => void;
}

export function ModelSelector({ value, onChange }: ModelSelectorProps) {
  const [mounted, setMounted] = useState(false);

  useEffect(() => setMounted(true), []);

  if (!mounted) return <select className="select select-bordered w-full max-w-xs" disabled />;

  return (
    <select
      className="select select-bordered w-full max-w-xs"
      value={value}
      onChange={(e) => onChange(e.target.value as ModelId)}
    >
      {Object.entries(modelLabels).map(([id, label]) => (
        <option key={id} value={id}>
          {label}
        </option>
      ))}
    </select>
  );
}

Build the chat interface

The chat component manages message state, calls the streaming endpoint, and renders tool invocations inline. The useChat hook from ai/react handles the streaming lifecycle.

// src/components/ChatInterface.tsx
'use client';

import { useChat } from 'ai/react';
import { ModelId } from '@/lib/models';
import { ModelSelector } from './ModelSelector';
import { useState } from 'react';

export function ChatInterface() {
  const [modelId, setModelId] = useState<ModelId>(() => {
    if (typeof window !== 'undefined') {
      return (localStorage.getItem('selectedModel') as ModelId) ?? 'gpt-4o';
    }
    return 'gpt-4o';
  });

  const { messages, input, handleInputChange, handleSubmit, isLoading, error, stop } = useChat({
    api: '/api/chat',
    body: { modelId },
    onFinish: (message) => {
      console.log('Finished:', message.role, message.content);
    },
    onError: (err) => {
      console.error('Chat error:', err);
    },
  });

  const handleModelChange = (newModelId: ModelId) => {
    localStorage.setItem('selectedModel', newModelId);
    setModelId(newModelId);
  };

  return (
    <div className="flex flex-col h-[calc(100vh-4rem)] p-4 gap-4">
      <div className="flex items-center justify-between">
        <h2 className="text-xl font-semibold">Multi-model chat</h2>
        <ModelSelector value={modelId} onChange={handleModelChange} />
      </div>

      <div className="flex-1 overflow-y-auto space-y-4">
        {messages.map((message) => (
          <div key={message.id} className={`flex gap-3 ${message.role === 'assistant' ? 'flex-col' : ''}`}>
            <div className={`chat-bubble ${message.role === 'user' ? 'chat-bubble-primary' : 'chat-bubble-base-100'}`}>
              {message.role === 'assistant' && message.toolInvocations?.length ? (
                <details className="text-sm">
                  <summary className="font-mono text-gray-500">Tool calls</summary>
                  <pre className="mt-1 p-2 bg-gray-100 rounded overflow-auto">
                    {JSON.stringify(message.toolInvocations, null, 2)}
                  </pre>
                </details>
              ) : null}
              <div className="whitespace-pre-wrap">{message.content}</div>
            </div>
          </div>
        ))}
        {isLoading && (
          <div className="chat-bubble chat-bubble-base-100 animate-pulse">Thinking…</div>
        )}
      </div>

      {error && (
        <div className="alert alert-error">{error.message}</div>
      )}

      <form onSubmit={handleSubmit} className="flex gap-2">
        <input
          className="input input-bordered flex-1"
          value={input}
          onChange={handleInputChange}
          placeholder="Ask about weather, or anything else…"
          disabled={isLoading}
        />
        {isLoading ? (
          <button type="button" className="btn btn-primary" onClick={stop}>Stop</button>
        ) : (
          <button type="submit" className="btn btn-primary">Send</button>
        )}
      </form>
    </div>
  );
}

Wire it into the page

// src/app/page.tsx
import { ChatInterface } from '@/components/ChatInterface';

export default function Home() {
  return (
    <main className="min-h-screen bg-base-100">
      <ChatInterface />
    </main>
  );
}

Update the layout to include Tailwind’s base styles and a minimal header:

// src/app/layout.tsx
import type { Metadata } from 'next';
import './globals.css';

export const metadata: Metadata = {
  title: 'Multi-model chat',
};

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body className="min-h-screen bg-base-100">
        <header className="navbar bg-base-100 border-b">
          <div className="flex-1">
            <a className="btn btn-ghost text-xl font-bold">Multi-model Chat</a>
          </div>
        </header>
        {children}
      </body>
    </html>
  );
}

Checkpoint: Run pnpm dev, open http://localhost:3000, select a model, and send a message. Try “What’s the weather in Denver?” — you should see a tool invocation followed by a formatted response. Switch models mid-conversation; the message history persists because useChat maintains it client-side.

Handling provider failures gracefully

The SDK’s streamText accepts an onError callback, but for automatic provider failover you need a wrapper that catches rate limits or 5xx errors and retries with a different model. Here’s a minimal pattern:

// src/lib/withFallback.ts
import { streamText, StreamTextResult, LanguageModel } from 'ai';

type FallbackModels = [LanguageModel, ...LanguageModel[]];

export async function streamWithFallback(
  models: FallbackModels,
  params: Parameters<typeof streamText>[0]
): Promise<StreamTextResult> {
  let lastError: Error;

  for (const model of models) {
    try {
      return streamText({ ...params, model });
    } catch (err) {
      lastError = err as Error;
      // Check for retryable conditions: rate limit, provider unavailable
      const isRetryable = err instanceof Error && (
        err.message.includes('rate limit') ||
        err.message.includes('503') ||
        err.message.includes('502') ||
        err.message.includes('overloaded')
      );
      if (!isRetryable) throw err;
      // Log and continue to next model
      console.warn(`Model ${model.modelId} failed, trying fallback:`, err);
    }
  }

  throw lastError!;
}

Then in your route:

// src/app/api/chat/route.ts (updated)
import { streamWithFallback } from '@/lib/withFallback';
import { models, getModel, ModelId } from '@/lib/models';
import { tools } from '@/lib/tools';

export async function POST(req: Request) {
  const { messages, modelId } = await req.json();

  const primary = getModel(modelId);
  // Fallback chain: prefer same capability tier
  const fallbacks: ModelId[] = modelId === 'mixtral-8x7b'
    ? ['gpt-4o', 'claude-3-5-sonnet']
    : modelId === 'claude-3-5-sonnet'
    ? ['gpt-4o', 'mixtral-8x7b']
    : ['claude-3-5-sonnet', 'mixtral-8x7b'];

  const modelChain = [primary, ...fallbacks.map(getModel)];

  const result = await streamWithFallback(modelChain, {
    messages,
    tools,
    maxTokens: 4096,
  });

  return result.toDataStreamResponse();
}

This keeps the user experience intact when a provider degrades. In production, you’d add exponential backoff, circuit breakers, and metrics — but the core pattern is five lines of logic.

Streaming tool results back to the client

The useChat hook automatically handles tool call/result cycles when your API returns the proper stream format. The streamText result’s toDataStreamResponse() emits data: {...} events for each step: tool-call, tool-result, text-delta, finish. The client renders them in order. No extra wiring needed.

If you need custom tool UI (e.g., a map for location lookups), extend the message type:

// src/lib/types.ts
import { ToolInvocation } from 'ai';

export interface ChatMessage {
  id: string;
  role: 'user' | 'assistant' | 'tool';
  content: string;
  toolInvocations?: ToolInvocation[];
  // Custom field for your UI
  toolDisplay?: React.ReactNode;
}

Then map over messages in ChatInterface and render toolDisplay when present.

Deployment notes

• Environment variables: Add ANTHROPIC_API_KEY, OPENAI_API_KEY, MISTRAL_API_KEY (or TOGETHER_API_KEY if using Together’s Mistral endpoint) to your hosting platform.
• Edge runtime: The route works on Edge if you avoid Node-only APIs. streamText supports Edge in AI SDK 3.1+.
• Caching: Responses aren’t cached by default. For read-heavy workloads, add export const dynamic = 'force-dynamic' to the route.
• Observability: Wrap streamText with a logger that emits modelId, latencyMs, tokenUsage, and finishReason to your metrics pipeline.

What this buys you

• Single code path for three architecturally different models (Anthropic, OpenAI, Mistral/Mixtral)
• Streaming by default with zero boilerplate
• Tool calling that works identically across providers
• Model switching without losing conversation context
• Fallback chain that degrades gracefully when a provider hits rate limits

The Vercel AI SDK’s provider abstraction is the leverage point. Once you’ve mapped your models into a registry, the rest is standard Next.js patterns. Swap in Cohere, Gemini, or a self-hosted Llama 3.1 by adding one line to models.ts — the chat interface, streaming logic, and fallback chain don’t change.