The Vercel AI SDK multi-model setup n4n.ai models approach lets you route requests across hundreds of models through one OpenAI-compatible endpoint. Instead of managing separate clients, API keys, and retry logic for each provider, you configure a single base URL and switch models by name. This guide walks through the complete implementation — from provider configuration to streaming responses and production-grade fallbacks.
Why consolidate behind one endpoint
Most teams start by adding providers one at a time: OpenAI for GPT-4, Anthropic for Claude, maybe Cohere or Mistral for specific tasks. Each addition means a new SDK, new authentication, new rate-limit handling, and new billing reconciliation. The cognitive load compounds fast.
An OpenAI-compatible gateway solves this by presenting a uniform interface. You send a chat completion request with model: "anthropic/claude-3.5-sonnet" or model: "meta-llama/llama-3.1-405b" and the gateway handles provider selection, authentication, and response normalization. Your application code stays the same regardless of which model actually serves the request.
The tradeoff is clear: you introduce a network hop and depend on the gateway’s uptime. In exchange, you eliminate provider-specific code paths and gain centralized observability. For most teams building user-facing LLM features, that tradeoff pays off within the first few integrations.
Install and configure the AI SDK
Start with the core packages. The AI SDK v4 uses a provider pattern where each model provider exports a createOpenAI compatible client.
npm install ai @ai-sdk/openai zodCreate a single gateway client that points to your endpoint. The base URL is the only configuration that changes between environments.
// lib/gateway.ts
import { createOpenAI } from '@ai-sdk/openai';
export const gateway = createOpenAI({
baseURL: process.env.GATEWAY_BASE_URL ?? 'https://api.n4n.ai/v1',
apiKey: process.env.GATEWAY_API_KEY,
// Optional: forward custom headers for routing hints
headers: {
'x-router-model-preference': 'cost', // or 'latency', 'quality'
},
});The x-router-model-preference header is a routing directive the gateway honors — it tells the load balancer to prefer cheaper models, lower latency, or highest quality when multiple models satisfy the request. This is useful for background jobs versus user-facing chat.
Define your model registry
Hardcoding model strings across your codebase creates maintenance pain. Centralize them in a registry with metadata so you can reason about capabilities, pricing, and fallbacks programmatically.
// lib/models.ts
export const models = {
// General purpose
'gpt-4o': { provider: 'openai', contextWindow: 128_000, supportsTools: true },
'gpt-4o-mini': { provider: 'openai', contextWindow: 128_000, supportsTools: true },
'claude-3.5-sonnet': { provider: 'anthropic', contextWindow: 200_000, supportsTools: true },
'claude-3.5-haiku': { provider: 'anthropic', contextWindow: 200_000, supportsTools: true },
// Reasoning heavy
'o1-preview': { provider: 'openai', contextWindow: 128_000, supportsTools: false },
'o1-mini': { provider: 'openai', contextWindow: 128_000, supportsTools: false },
// Open weight models
'meta-llama/llama-3.1-405b': { provider: 'meta', contextWindow: 128_000, supportsTools: true },
'meta-llama/llama-3.1-70b': { provider: 'meta', contextWindow: 128_000, supportsTools: true },
'mistral-large': { provider: 'mistral', contextWindow: 128_000, supportsTools: true },
// Specialized
'gemini-1.5-pro': { provider: 'google', contextWindow: 2_000_000, supportsTools: true },
'command-r-plus': { provider: 'cohere', contextWindow: 128_000, supportsTools: true },
} as const;
export type ModelId = keyof typeof models;
export function getModel(id: ModelId) {
return models[id];
}
export function listModels(): ModelId[] {
return Object.keys(models) as ModelId[];
}This registry becomes your source of truth for feature flags, UI model selectors, and automatic fallback chains.
Build a model resolver with fallbacks
Production systems need graceful degradation. When your primary model is rate-limited or degraded, you want automatic fallback without surfacing errors to users. The gateway can handle this at the infrastructure level, but application-level fallback gives you control over the policy.
// lib/model-resolver.ts
import { gateway } from './gateway';
import { models, ModelId, getModel } from './models';
interface ResolveOptions {
preferred?: ModelId[];
requireTools?: boolean;
maxContext?: number;
preferCost?: boolean;
}
export function resolveModel(options: ResolveOptions = {}): ModelId {
const { preferred = ['gpt-4o-mini', 'claude-3.5-haiku'], requireTools, maxContext, preferCost } = options;
let candidates = preferred.filter(id => {
const model = getModel(id);
if (requireTools && !model.supportsTools) return false;
if (maxContext && model.contextWindow < maxContext) return false;
return true;
});
if (candidates.length === 0) {
// Fall back to all models matching constraints
candidates = Object.keys(models).filter(id => {
const model = getModel(id);
if (requireTools && !model.supportsTools) return false;
if (maxContext && model.contextWindow < maxContext) return false;
return true;
}) as ModelId[];
}
if (candidates.length === 0) {
throw new Error('No models match the required capabilities');
}
// Sort by cost if requested (cheapest first)
if (preferCost) {
candidates.sort((a, b) => estimateCost(a) - estimateCost(b));
}
return candidates[0];
}
// Rough cost estimates per 1k tokens (input + output blended)
// Update these from your provider pricing page periodically
function estimateCost(modelId: ModelId): number {
const costs: Record<ModelId, number> = {
'gpt-4o': 15,
'gpt-4o-mini': 0.6,
'claude-3.5-sonnet': 9,
'claude-3.5-haiku': 0.8,
'o1-preview': 60,
'o1-mini': 12,
'meta-llama/llama-3.1-405b': 3,
'meta-llama/llama-3.1-70b': 0.9,
'mistral-large': 4,
'gemini-1.5-pro': 3.5,
'command-r-plus': 3,
};
return costs[modelId] ?? 10;
}
export function createModelClient(modelId: ModelId) {
return gateway(modelId);
}The resolver encodes your operational preferences: try fast/cheap models first, require tool support for agent workflows, respect context window needs, and optionally optimize for cost. The gateway still provides infrastructure-level fallback if the resolved model fails, but this layer lets you express intent explicitly.
Streaming chat with tool calls
The AI SDK’s streamText handles the heavy lifting. Pass the resolved model client and your tools — the SDK manages the request/response cycle, tool execution, and streaming delta emission.
// app/api/chat/route.ts
import { streamText } from 'ai';
import { resolveModel, createModelClient } from '@/lib/model-resolver';
import { z } from 'zod';
const tools = {
getWeather: {
parameters: z.object({
location: z.string().describe('City and state, e.g. "San Francisco, CA"'),
unit: z.enum(['celsius', 'fahrenheit']).default('fahrenheit'),
}),
execute: async ({ location, unit }) => {
// Your weather API call here
return { temperature: 72, condition: 'sunny', unit };
},
},
searchWeb: {
parameters: z.object({
query: z.string(),
maxResults: z.number().default(5),
}),
execute: async ({ query, maxResults }) => {
// Your search API call here
return [{ title: 'Result 1', url: 'https://example.com', snippet: '...' }];
},
},
};
export async function POST(req: Request) {
const { messages, model: requestedModel, preferCost } = await req.json();
const modelId = resolveModel({
preferred: requestedModel ? [requestedModel] : undefined,
requireTools: true,
preferCost,
});
const model = createModelClient(modelId);
const result = streamText({
model,
messages,
tools,
maxSteps: 5, // Prevent infinite tool loops
temperature: 0.3,
// Forward cache-control hints to the gateway
headers: {
'x-cache-control': 'max-age=300, stale-while-revalidate=600',
},
onError: (error) => {
console.error('Stream error:', error);
// Gateway will retry on 5xx, but log for observability
},
});
return result.toDataStreamResponse({
// Send model metadata to the client for UI display
sendReasoning: true,
sendUsage: true,
});
}Key points in this implementation:
maxSteps: 5prevents runaway tool loops — a common production issue when models get stuck in recursive calls- The
headersobject forwards cache-control hints the gateway respects for response caching sendUsage: truestreams token usage back to the client for real-time cost display- The resolver runs on every request, so model selection adapts to runtime conditions
Handling provider-specific quirks
Even with a unified interface, models behave differently. Three areas bite teams repeatedly:
Tool call formats. OpenAI uses function calling, Anthropic uses tool_use blocks, and some open-weight models expect JSON in the response body. The AI SDK normalizes most of this, but you’ll still hit edge cases with parallel tool calls or malformed arguments. Always validate tool inputs with Zod schemas and handle parse failures gracefully.
// lib/tool-validation.ts
import { z } from 'zod';
export function validateToolArgs<T extends z.ZodTypeAny>(
schema: T,
args: unknown
): z.infer<T> {
const result = schema.safeParse(args);
if (!result.success) {
throw new Error(`Invalid tool arguments: ${result.error.message}`);
}
return result.data;
}Context window overflow. Models have different limits. The gateway truncates automatically, but silent truncation loses critical context. Implement a pre-flight check:
// lib/context-guard.ts
import { getModel, ModelId } from './models';
import { countTokens } from '@/lib/tokenizer'; // Use tiktoken or similar
export function guardContextWindow(
messages: Array<{ role: string; content: string }>,
modelId: ModelId,
reserveTokens = 4096
): { fits: boolean; used: number; available: number } {
const model = getModel(modelId);
const used = messages.reduce((sum, m) => sum + countTokens(m.content), 0);
const available = model.contextWindow - reserveTokens;
return {
fits: used <= available,
used,
available,
};
}Call this before streaming and either summarize history or reject the request with a clear error.
Streaming delimiter differences. Some providers stream tool calls as separate chunks, others inline them. The AI SDK handles reassembly, but if you’re parsing raw streams for custom UIs, test each model family. The safest path is letting the SDK manage the stream and consuming dataStreamResponse.
Observability and cost tracking
You can’t optimize what you don’t measure. The gateway returns usage metadata in response headers and stream events. Capture this at the edge.
// lib/usage-tracker.ts
interface UsageEvent {
model: string;
promptTokens: number;
completionTokens: number;
totalTokens: number;
estimatedCost: number;
latencyMs: number;
timestamp: Date;
requestId: string;
}
const usageBuffer: UsageEvent[] = [];
const FLUSH_INTERVAL = 10_000;
const BATCH_SIZE = 100;
export function recordUsage(event: UsageEvent) {
usageBuffer.push(event);
if (usageBuffer.length >= BATCH_SIZE) flush();
}
setInterval(flush, FLUSH_INTERVAL);
async function flush() {
if (usageBuffer.length === 0) return;
const batch = usageBuffer.splice(0, BATCH_SIZE);
try {
await fetch('/api/internal/usage', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(batch),
});
} catch (e) {
// Re-queue on failure
usageBuffer.unshift(...batch);
console.error('Usage flush failed:', e);
}
}Wire this into your streaming response:
// In your route handler
const result = streamText({
// ... config
onFinish: ({ usage, finishReason, latencyMs }) => {
recordUsage({
model: modelId,
promptTokens: usage.promptTokens,
completionTokens: usage.completionTokens,
totalTokens: usage.totalTokens,
estimatedCost: estimateCost(modelId) * usage.totalTokens / 1000,
latencyMs,
timestamp: new Date(),
requestId: crypto.randomUUID(),
});
},
});This gives you per-request cost, latency percentiles by model, and failure rates — essential for capacity planning and model selection decisions.
Common pitfalls
Assuming all models support the same features. o1-preview doesn’t support tools. gemini-1.5-pro has a 2M token window but different safety filtering. Your resolver must encode these constraints, not just model names.
Ignoring gateway rate limits. The gateway aggregates across providers, but its own limits apply. Implement client-side backoff with Retry-After header respect:
// lib/with-retry.ts
export async function withRetry<T>(
fn: () => Promise<T>,
maxRetries = 3
): Promise<T> {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (error: any) {
if (attempt === maxRetries) throw error;
const retryAfter = error.response?.headers?.['retry-after'];
const delay = retryAfter ? parseInt(retryAfter) * 1000 : Math.min(1000 * 2 ** attempt, 30_000);
await new Promise(r => setTimeout(r, delay + Math.random() * 500));
}
}
throw new Error('Max retries exceeded');
}Hardcoding model IDs in UI components. Your model selector should fetch available models from an endpoint that reads the registry, not from a static list. This lets you roll out new models without frontend deployments.
// app/api/models/route.ts
import { listModels, getModel } from '@/lib/models';
export async function GET() {
return Response.json(
listModels().map(id => ({
id,
...getModel(id),
}))
);
}Forgetting streaming timeouts. Long-running streams (especially with tools) can exceed load balancer timeouts. Configure your platform for 5+ minute timeouts on chat endpoints, or implement a heartbeat mechanism.
Testing strategy
Unit test your resolver logic with a matrix of constraints. Integration test against the gateway with a small set of representative models — don’t test all 240+. Use a test API key with a strict budget.
// lib/model-resolver.test.ts
import { resolveModel } from './model-resolver';
describe('resolveModel', () => {
it('prefers cost when requested', () => {
const model = resolveModel({ preferCost: true, requireTools: true });
expect(['gpt-4o-mini', 'claude-3.5-haiku', 'meta-llama/llama-3.1-70b']).toContain(model);
});
it('excludes non-tool models when tools required', () => {
const model = resolveModel({ requireTools: true, preferred: ['o1-preview', 'gpt-4o'] });
expect(model).toBe('gpt-4o');
});
it('respects context window minimum', () => {
const model = resolveModel({ maxContext: 1_000_000 });
expect(model).toBe('gemini-1.5-pro');
});
});Load test your streaming endpoint with concurrent tool-heavy requests. Watch for connection pool exhaustion in the gateway client — the default undici agent may need tuning:
// lib/gateway.ts (addition)
import { Agent } from 'undici';
export const gateway = createOpenAI({
baseURL: process.env.GATEWAY_BASE_URL ?? 'https://api.n4n.ai/v1',
apiKey: process.env.GATEWAY_API_KEY,
fetch: (url, options) => {
const agent = new Agent({
connections: 100,
keepAliveTimeout: 30_000,
keepAliveMaxTimeout: 300_000,
});
return fetch(url, { ...options, dispatcher: agent });
},
});Rolling out model changes
Treat model upgrades like database migrations. Add the new model to your registry, deploy the resolver update, then gradually shift traffic using a feature flag or weighted routing header.
// lib/rollout.ts
export function getRolloutModel(userId: string): ModelId {
// Deterministic hash for consistent assignment
const hash = hashString(userId);
const bucket = hash % 100;
if (bucket < 5) return 'gpt-4o'; // 5% canary
if (bucket < 15) return 'claude-3.5-sonnet'; // 10% canary
return 'gpt-4o-mini'; // 85% stable
}Monitor error rates, latency, and user feedback metrics per model bucket before promoting. The gateway’s per-token metering makes this analysis straightforward — you already have the data.
What this buys you
One endpoint. One authentication scheme. One retry policy. One observability stack. Your application code expresses intent — “give me a tool-capable model optimized for cost” — and the infrastructure delivers. When a new model launches, you add it to the registry and it’s available everywhere. When a provider has an outage, the gateway routes around it without code changes.
The Vercel AI SDK multi-model setup n4n.ai models pattern scales from prototype to production without rewrites. Start with the resolver, add observability early, and treat model selection as a configurable policy — not hardcoded logic.