Per-request model overrides in the Vercel AI SDK

The Vercel AI SDK makes it straightforward to swap models per request, but the documentation scatters the relevant pieces across multiple pages. This guide walks through a complete, production-ready pattern for vercel ai sdk per-request model override logic — from request validation through provider fallback — so you can route each conversation turn to the right model without restarting your server.

Step 1: Define your model registry

Start with a single source of truth for the models your application supports. Keep it in a dedicated module so both your API routes and any admin tooling import the same definitions.

// lib/models.ts
export type ModelId = 
  | 'gpt-4o'
  | 'gpt-4o-mini'
  | 'claude-3-5-sonnet'
  | 'claude-3-haiku'
  | 'llama-3.1-70b'
  | 'llama-3.1-8b';

export interface ModelSpec {
  id: ModelId;
  provider: 'openai' | 'anthropic' | 'groq' | 'together';
  displayName: string;
  maxTokens: number;
  supportsTools: boolean;
  supportsVision: boolean;
  costPer1kInputTokens: number;
  costPer1kOutputTokens: number;
}

export const MODEL_REGISTRY: Record<ModelId, ModelSpec> = {
  'gpt-4o': {
    id: 'gpt-4o',
    provider: 'openai',
    displayName: 'GPT-4o',
    maxTokens: 128_000,
    supportsTools: true,
    supportsVision: true,
    costPer1kInputTokens: 0.005,
    costPer1kOutputTokens: 0.015,
  },
  'gpt-4o-mini': {
    id: 'gpt-4o-mini',
    provider: 'openai',
    displayName: 'GPT-4o Mini',
    maxTokens: 128_000,
    supportsTools: true,
    supportsVision: true,
    costPer1kInputTokens: 0.00015,
    costPer1kOutputTokens: 0.0006,
  },
  'claude-3-5-sonnet': {
    id: 'claude-3-5-sonnet',
    provider: 'anthropic',
    displayName: 'Claude 3.5 Sonnet',
    maxTokens: 200_000,
    supportsTools: true,
    supportsVision: true,
    costPer1kInputTokens: 0.003,
    costPer1kOutputTokens: 0.015,
  },
  'claude-3-haiku': {
    id: 'claude-3-haiku',
    provider: 'anthropic',
    displayName: 'Claude 3 Haiku',
    maxTokens: 200_000,
    supportsTools: true,
    supportsVision: true,
    costPer1kInputTokens: 0.00025,
    costPer1kOutputTokens: 0.00125,
  },
  'llama-3.1-70b': {
    id: 'llama-3.1-70b',
    provider: 'groq',
    displayName: 'Llama 3.1 70B',
    maxTokens: 128_000,
    supportsTools: true,
    supportsVision: false,
    costPer1kInputTokens: 0.00059,
    costPer1kOutputTokens: 0.00079,
  },
  'llama-3.1-8b': {
    id: 'llama-3.1-8b',
    provider: 'groq',
    displayName: 'Llama 3.1 8B',
    maxTokens: 128_000,
    supportsTools: true,
    supportsVision: false,
    costPer1kInputTokens: 0.00005,
    costPer1kOutputTokens: 0.00008,
  },
};

export function getModelSpec(id: string): ModelSpec | undefined {
  return MODEL_REGISTRY[id as ModelId];
}

export function listAvailableModels(): ModelSpec[] {
  return Object.values(MODEL_REGISTRY);
}

This registry drives validation, cost estimation, and capability checks downstream. Add or remove entries here without touching route handlers.

Step 2: Build a provider-agnostic model factory

The AI SDK’s createOpenAI, createAnthropic, and similar factories return provider-specific clients. Wrap them in a single function that resolves the correct client from your registry.

// lib/model-factory.ts
import { createOpenAI } from '@ai-sdk/openai';
import { createAnthropic } from '@ai-sdk/anthropic';
import { createGroq } from '@ai-sdk/groq';
import { getModelSpec, ModelSpec } from './models';

const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY });
const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const groq = createGroq({ apiKey: process.env.GROQ_API_KEY });

export function getModelClient(spec: ModelSpec) {
  switch (spec.provider) {
    case 'openai':
      return openai(spec.id);
    case 'anthropic':
      return anthropic(spec.id);
    case 'groq':
      return groq(spec.id);
    default:
      throw new Error(`Unsupported provider: ${spec.provider}`);
  }
}

export function validateModelRequest(modelId: string, requireTools = false, requireVision = false): ModelSpec {
  const spec = getModelSpec(modelId);
  if (!spec) {
    throw new Error(`Unknown model: ${modelId}`);
  }
  if (requireTools && !spec.supportsTools) {
    throw new Error(`Model ${modelId} does not support tool calling`);
  }
  if (requireVision && !spec.supportsVision) {
    throw new Error(`Model ${modelId} does not support vision`);
  }
  return spec;
}

The factory centralizes provider initialization. If you later add Together AI or Bedrock, you only touch this file.

Step 3: Create the chat route with per-request override

Now wire the factory into a Next.js App Router route handler. The key insight: the model identifier comes from the request body, not a config file.

// app/api/chat/route.ts
import { streamText, CoreMessage } from 'ai';
import { getModelClient, validateModelRequest } from '@/lib/model-factory';
import { getModelSpec } from '@/lib/models';
import { z } from 'zod';

const ChatRequestSchema = z.object({
  messages: z.array(z.object({
    role: z.enum(['user', 'assistant', 'system', 'tool']),
    content: z.union([z.string(), z.array(z.any())]),
  })),
  model: z.string().min(1),
  temperature: z.number().min(0).max(2).optional(),
  maxTokens: z.number().min(1).max(128_000).optional(),
  tools: z.record(z.any()).optional(),
  requireTools: z.boolean().optional(),
  requireVision: z.boolean().optional(),
});

export async function POST(req: Request) {
  let body: z.infer<typeof ChatRequestSchema>;
  
  try {
    const json = await req.json();
    body = ChatRequestSchema.parse(json);
  } catch (err) {
    return new Response(JSON.stringify({ error: 'Invalid request body' }), {
      status: 400,
      headers: { 'Content-Type': 'application/json' },
    });
  }

  const { messages, model: modelId, temperature, maxTokens, tools, requireTools, requireVision } = body;

  let spec;
  try {
    spec = validateModelRequest(modelId, requireTools, requireVision);
  } catch (err) {
    return new Response(JSON.stringify({ error: err instanceof Error ? err.message : 'Invalid model' }), {
      status: 400,
      headers: { 'Content-Type': 'application/json' },
    });
  }

  const model = getModelClient(spec);

  const systemPrompt = `You are a helpful assistant. Current model: ${spec.displayName} (${spec.provider}).`;

  try {
    const result = streamText({
      model,
      messages: [
        { role: 'system', content: systemPrompt },
        ...messages.map((m): CoreMessage => ({
          role: m.role,
          content: m.content,
        })),
      ],
      temperature: temperature ?? 0.7,
      maxTokens: maxTokens ?? spec.maxTokens,
      tools,
    });

    return result.toDataStreamResponse({
      headers: {
        'x-model-used': spec.id,
        'x-model-provider': spec.provider,
      },
    });
  } catch (err) {
    console.error('Stream error:', err);
    return new Response(JSON.stringify({ error: 'Generation failed' }), {
      status: 500,
      headers: { 'Content-Type': 'application/json' },
    });
  }
}

The route validates the requested model against your registry, enforces capability requirements, and streams the response with headers identifying which model actually served the request — useful for debugging and analytics.

Step 4: Add provider fallback for resilience

Production systems need graceful degradation when a provider hits rate limits or goes down. Extend the factory with a fallback chain per model.

// lib/model-factory.ts (extended)
import { LanguageModelV1 } from 'ai';

interface FallbackChain {
  primary: ModelSpec;
  fallbacks: ModelSpec[];
}

const FALLBACK_CHAINS: Record<string, FallbackChain> = {
  'gpt-4o': {
    primary: MODEL_REGISTRY['gpt-4o'],
    fallbacks: [MODEL_REGISTRY['claude-3-5-sonnet'], MODEL_REGISTRY['llama-3.1-70b']],
  },
  'gpt-4o-mini': {
    primary: MODEL_REGISTRY['gpt-4o-mini'],
    fallbacks: [MODEL_REGISTRY['claude-3-haiku'], MODEL_REGISTRY['llama-3.1-8b']],
  },
  'claude-3-5-sonnet': {
    primary: MODEL_REGISTRY['claude-3-5-sonnet'],
    fallbacks: [MODEL_REGISTRY['gpt-4o'], MODEL_REGISTRY['llama-3.1-70b']],
  },
  'claude-3-haiku': {
    primary: MODEL_REGISTRY['claude-3-haiku'],
    fallbacks: [MODEL_REGISTRY['gpt-4o-mini'], MODEL_REGISTRY['llama-3.1-8b']],
  },
  'llama-3.1-70b': {
    primary: MODEL_REGISTRY['llama-3.1-70b'],
    fallbacks: [MODEL_REGISTRY['gpt-4o'], MODEL_REGISTRY['claude-3-5-sonnet']],
  },
  'llama-3.1-8b': {
    primary: MODEL_REGISTRY['llama-3.1-8b'],
    fallbacks: [MODEL_REGISTRY['gpt-4o-mini'], MODEL_REGISTRY['claude-3-haiku']],
  },
};

export function getModelWithFallback(modelId: string): LanguageModelV1 {
  const chain = FALLBACK_CHAINS[modelId];
  if (!chain) {
    const spec = getModelSpec(modelId);
    if (!spec) throw new Error(`Unknown model: ${modelId}`);
    return getModelClient(spec);
  }

  const models = [chain.primary, ...chain.fallbacks].map(getModelClient);
  
  return {
    ...models[0],
    provider: 'fallback-chain',
    modelId: chain.primary.id,
    async doGenerate(options) {
      let lastError: Error | null = null;
      
      for (const model of models) {
        try {
          return await model.doGenerate(options);
        } catch (err) {
          lastError = err instanceof Error ? err : new Error(String(err));
          const isRetryable = err instanceof Error && (
            err.message.includes('rate limit') ||
            err.message.includes('503') ||
            err.message.includes('502') ||
            err.message.includes('timeout')
          );
          if (!isRetryable) break;
          console.warn(`Model ${model.modelId} failed, trying fallback:`, lastError.message);
        }
      }
      
      throw lastError ?? new Error('All fallbacks exhausted');
    },
    async doStream(options) {
      let lastError: Error | null = null;
      
      for (const model of models) {
        try {
          return await model.doStream(options);
        } catch (err) {
          lastError = err instanceof Error ? err : new Error(String(err));
          const isRetryable = err instanceof Error && (
            err.message.includes('rate limit') ||
            err.message.includes('503') ||
            err.message.includes('502') ||
            err.message.includes('timeout')
          );
          if (!isRetryable) break;
          console.warn(`Model ${model.modelId} failed, trying fallback:`, lastError.message);
        }
      }
      
      throw lastError ?? new Error('All fallbacks exhausted');
    },
  } satisfies LanguageModelV1;
}

Update the route handler to use getModelWithFallback instead of getModelClient. The fallback chain preserves the original model’s capabilities — if the primary supports tools, the fallbacks do too, because you defined them that way in the registry.

Step 5: Wire the client-side model selector

A per-request override is useless without a way to choose the model. Here’s a minimal React component that posts to your route.

// components/ModelSelector.tsx
'use client';

import { useState } from 'react';
import { listAvailableModels, ModelSpec } from '@/lib/models';

interface ModelSelectorProps {
  selectedModel: string;
  onChange: (modelId: string) => void;
  disabled?: boolean;
}

export function ModelSelector({ selectedModel, onChange, disabled }: ModelSelectorProps) {
  const models = listAvailableModels();
  
  return (
    <select
      value={selectedModel}
      onChange={(e) => onChange(e.target.value)}
      disabled={disabled}
      className="px-3 py-2 border rounded-md bg-white text-sm"
    >
      {models.map((model: ModelSpec) => (
        <option key={model.id} value={model.id}>
          {model.displayName} ({model.provider})
        </option>
      ))}
    </select>
  );
}

// components/ChatInterface.tsx
'use client';

import { useState, useRef, useEffect } from 'react';
import { ModelSelector } from './ModelSelector';
import { listAvailableModels } from '@/lib/models';

interface Message {
  role: 'user' | 'assistant';
  content: string;
}

export function ChatInterface() {
  const [messages, setMessages] = useState<Message[]>([]);
  const [input, setInput] = useState('');
  const [selectedModel, setSelectedModel] = useState(listAvailableModels()[0].id);
  const [isStreaming, setIsStreaming] = useState(false);
  const [usedModel, setUsedModel] = useState<string | null>(null);
  const abortRef = useRef<AbortController | null>(null);
  const messagesEndRef = useRef<HTMLDivElement>(null);

  const scrollToBottom = () => {
    messagesEndRef.current?.scrollIntoView({ behavior: 'smooth' });
  };

  useEffect(() => {
    scrollToBottom();
  }, [messages]);

  const handleSubmit = async (e: React.FormEvent) => {
    e.preventDefault();
    if (!input.trim() || isStreaming) return;

    const userMessage = { role: 'user' as const, content: input };
    setMessages((prev) => [...prev, userMessage]);
    setInput('');
    setIsStreaming(true);
    setUsedModel(null);

    abortRef.current = new AbortController();

    try {
      const response = await fetch('/api/chat', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          messages: [...messages, userMessage],
          model: selectedModel,
          temperature: 0.7,
        }),
        signal: abortRef.current.signal,
      });

      if (!response.ok) {
        const error = await response.json();
        throw new Error(error.error ?? 'Request failed');
      }

      const modelUsed = response.headers.get('x-model-used');
      if (modelUsed) setUsedModel(modelUsed);

      const reader = response.body?.getReader();
      const decoder = new TextDecoder();
      let assistantContent = '';

      if (reader) {
        setMessages((prev) => [...prev, { role: 'assistant', content: '' }]);
        
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          
          const chunk = decoder.decode(value);
          const lines = chunk.split('\n');
          
          for (const line of lines) {
            if (line.startsWith('0:')) {
              const content = JSON.parse(line.slice(2));
              assistantContent += content;
              setMessages((prev) => {
                const next = [...prev];
                next[next.length - 1] = { role: 'assistant', content: assistantContent };
                return next;
              });
            }
          }
        }
      }
    } catch (err) {
      if (err instanceof Error && err.name !== 'AbortError') {
        console.error('Chat error:', err);
        setMessages((prev) => [...prev, { role: 'assistant', content: `Error: ${err.message}` }]);
      }
    } finally {
      setIsStreaming(false);
      abortRef.current = null;
    }
  };

  const handleStop = () => {
    abortRef.current?.abort();
    setIsStreaming(false);
  };

  return (
    <div className="flex flex-col h-[600px] border rounded-lg overflow-hidden">
      <div className="p-3 border-b flex gap-2 items-center">
        <ModelSelector
          selectedModel={selectedModel}
          onChange={setSelectedModel}
          disabled={isStreaming}
        />
        {usedModel && (
          <span className="text-xs text-gray-500 ml-auto">
            Served by: {usedModel}
          </span>
        )}
      </div>
      
      <div className="flex-1 overflow-y-auto p-4 space-y-4">
        {messages.map((msg, idx) => (
          <div key={idx} className={`flex ${msg.role === 'user' ? 'justify-end' : 'justify-start'}`}>
            <div
              className={`max-w-[70%] p-3 rounded-lg ${
                msg.role === 'user' 
                  ? 'bg-blue-100 rounded-br-none' 
                  : 'bg-gray-100 rounded-bl-none'
              }`}
            >
              <pre className="whitespace-pre-wrap">{msg.content}</pre>
            </div>
          </div>
        ))}
        <div ref={messagesEndRef} />
      </div>

      <form onSubmit={handleSubmit} className="p-3 border-t flex gap-2">
        <input
          type="text"
          value={input}
          onChange={(e) => setInput(e.target.value)}
          placeholder="Type a message..."
          className="flex-1 px-3 py-2 border rounded-md"
          disabled={isStreaming}
        />
        <button
          type="submit"
          disabled={!input.trim() || isStreaming}
          className="px-4 py-2 bg-blue-600 text-white rounded-md disabled:opacity-50"
        >
          {isStreaming ? 'Streaming...' : 'Send'}
        </button>
        {isStreaming && (
          <button
            type="button"
            onClick={handleStop}
            className="px-4 py-2 bg-red-600 text-white rounded-md"
          >
            Stop
          </button>
        )}
      </form>
    </div>
  );
}

The client displays which model actually served the response via the x-model-used header — confirming the override worked end to end.

Step 6: Verify the implementation

Run the development server and test each path:

npm run dev

1. Basic override: Select “GPT-4o Mini” in the dropdown, send a message. Verify the response header x-model-used: gpt-4o-mini appears in the Network tab and the UI shows “Served by: gpt-4o-mini”.

2. Capability enforcement: Add requireTools: true to the request body and select “Llama 3.1 8B” (which supports tools). Then try with a model that doesn’t — the route should return 400 before calling any provider.

3. Fallback trigger: Temporarily invalidate your OpenAI API key, select “GPT-4o”, and send a request. The response should come from Claude 3.5 Sonnet (the first fallback) and the header should read x-model-used: claude-3-5-sonnet.

4. Cost tracking: The registry includes per-token pricing. Extend the route to log usage.promptTokens * spec.costPer1kInputTokens / 1000 per request for real-time cost dashboards.

Step 7: Add routing directives for advanced control

If you operate a gateway that sits in front of multiple providers — like n4n.ai — you can pass routing hints through the request without changing the client SDK. The AI SDK forwards extra headers and body fields to the provider.

// app/api/chat/route.ts (add to request body parsing)
const ExtendedChatRequestSchema = ChatRequestSchema.extend({
  routing: z.object({
    preferProvider: z.enum(['openai', 'anthropic', 'groq']).optional(),
    maxLatencyMs: z.number().optional(),
    requireCache: z.boolean().optional(),
  }).optional(),
});

// Inside the handler, after validating the model:
if (body.routing?.preferProvider && body.routing.preferProvider !== spec.provider) {
  // Find an alternative model with the preferred provider
  const alternatives = listAvailableModels().filter(
    (m) => m.provider === body.routing!.preferProvider && m.supportsTools === spec.supportsTools
  );
  if (alternatives.length > 0) {
    spec = alternatives[0];
  }
}

This lets callers express preferences (“use Anthropic if available”) while your registry remains the authority on capability matching.

Common pitfalls

Mismatched tool schemas across providers: OpenAI and Anthropic handle tool definitions differently. If you pass tools in the request, normalize them per-provider in the factory or use the AI SDK’s tool helper which abstracts some differences.

Streaming fallback mid-response: The fallback implementation above only retries on complete failure. If a stream errors halfway through, you’d need to restart from the beginning with the fallback model — there’s no seamless mid-stream handoff.

Context window overflow: The registry stores maxTokens but the route doesn’t truncate history. Add a preprocessing step that estimates tokens and drops oldest messages until the prompt fits.

Caching model responses: The AI SDK supports cacheControl in tool results and messages. If you enable provider caching (Anthropic prompt caching, OpenAI cached completions), include the appropriate headers and verify the x-model-used header still reflects the actual serving model.

Next steps

• Persist the selected model per user in a cookie or database for sticky sessions
• Add request logging with model, latency, token counts, and cost to your observability stack
• Build an admin endpoint that returns listAvailableModels() with real-time provider health status
• Implement canary routing: send 5% of traffic to a new model and compare quality metrics

The pattern scales. Whether you’re switching between OpenAI and Anthropic today or adding a fine-tuned Llama on dedicated infrastructure tomorrow, the registry-factory-route chain keeps the complexity contained.