Cost-aware model routing in Vercel AI SDK apps

Most LLM bills spike because every request hits the most expensive model by default. Implementing vercel ai sdk cost-aware model routing lets you match model capability to task difficulty, cutting spend without measurable quality loss. This guide walks through a concrete routing layer you can ship this week.

Measure where tokens actually go

Before writing any routing logic, pull a week of usage logs. Group by endpoint or feature, not by model—you haven’t routed yet. The goal is to find requests that are high-volume but low-complexity: autocomplete, classification, short summaries, template filling.

A typical SaaS app burns 70–90% of tokens on trivial generation that a small model handles fine. Premium models cost an order of magnitude more per token. You can’t argue about routing until you see that distribution.

If you’re already on an OpenAI-compatible gateway, the usage field in each response is enough. Vercel AI SDK exposes it directly:

import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';

const { text, usage } = await generateText({
  model: openai('gpt-4o'),
  prompt: 'Summarize this support ticket',
});
console.log(usage.promptTokens, usage.completionTokens);

Log these to your analytics sink daily.

Tag requests by task type

Routing needs a signal. The cleanest signal is the calling code itself: a summarizeTicket function knows it’s doing summarization. Don’t try to build a general “difficulty classifier” model at first—that’s a second LLM call that eats the savings.

Define an enum or string union for task classes:

type TaskClass = 'classify' | 'extract' | 'summarize' | 'reason' | 'chat';

interface RoutedRequest {
  task: TaskClass;
  prompt: string;
  maxTokens?: number;
}

If you must infer complexity dynamically, use cheap heuristics: prompt length, presence of code blocks, or a keyword blocklist. Example:

function heuristicClass(prompt: string): TaskClass {
  if (prompt.length < 120) return 'classify';
  if (/```/.test(prompt)) return 'reason';
  return 'summarize';
&#125;

This is imperfect but transparent and free.

Implement the router in Vercel AI SDK

The core of vercel ai sdk cost-aware model routing is a thin function that maps TaskClass to a model instance. Keep the mapping in one place so you can adjust pricing tiers without hunting through handlers.

import { generateText } from 'ai';
import { createOpenAI } from '@ai-sdk/openai';

const provider = createOpenAI({ apiKey: process.env.OPENAI_API_KEY });

const MODEL_MAP = {
  classify: provider('gpt-4o-mini'),
  extract: provider('gpt-4o-mini'),
  summarize: provider('gpt-4o-mini'),
  reason: provider('gpt-4o'),
  chat: provider('gpt-4o'),
} as const;

export async function routedGenerate(req: RoutedRequest) {
  const model = MODEL_MAP[req.task];
  const { text, usage } = await generateText({
    model,
    prompt: req.prompt,
    maxTokens: req.maxTokens ?? 512,
  });
  return { text, usage, model: req.task };
}

Streaming and structured output

If you use streamText or generateObject, the same map works. For structured extraction, small models often suffice with a strict schema:

import { generateObject } from 'ai';
import { z } from 'zod';

const schema = z.object({ sentiment: z.enum(['pos', 'neg', 'neu']) });

const { object } = await generateObject({
  model: MODEL_MAP.classify,
  schema,
  prompt: req.prompt,
});

Swapping MODEL_MAP.classify to a cheaper ID later is a one-line change.

Use a single gateway for metering and fallback

Maintaining per-provider API keys, base URLs, and fallback logic inside your app is operational debt. Pointing the SDK at a gateway such as n4n.ai gives you a single OpenAI-compatible endpoint that addresses 240+ models, automatic fallback when a provider is rate-limited, and per-token usage metering out of the box.

const gateway = createOpenAI({
  baseURL: 'https://api.n4n.ai/v1',
  apiKey: process.env.N4N_API_KEY,
});

const MODEL_MAP = {
  classify: gateway('gpt-4o-mini'),
  reason: gateway('claude-3-5-sonnet'),
} as const;

The client routing directive is just the model string; the gateway forwards cache-control hints to the upstream provider. You stop writing retry loops for 429s.

Set hard limits and alerting

Routing reduces average cost but doesn’t bound worst case. A bug that loops reason tasks will still blow the budget. Enforce a per-request maxTokens and a daily token cap in your gateway or a middleware:

let dailyTokens = 0;
export async function routedGenerate(req: RoutedRequest) {
  if (dailyTokens > 2_000_000) throw new Error('daily token budget exceeded');
  const { text, usage } = await generateText({ /* ... */ });
  dailyTokens += usage.promptTokens + usage.completionTokens;
  return text;
}

Alert at 80% of budget to a Slack webhook. Don’t rely on the monthly invoice.

Tradeoffs and pitfalls

Latency vs cost

Small models are faster, but routing adds a classification step. If you use a heuristic, that’s microseconds. If you call a model to classify, you’ve spent tokens to save tokens—rarely worth it. Keep the decision local.

Quality cliffs

Some tasks look simple but aren’t. Extraction from messy PDFs fails silently on mini models, causing user retries that cost more than using a strong model once. Roll out routing behind a flag and compare output validation rates for two weeks.

Cache misses

Provider prompt caches (e.g., OpenAI’s cache_control) save money on repeated system prompts. If your router changes the model per request, the cache namespace changes. A gateway that forwards cache-control hints preserves this; ad-hoc base URL swaps may not. Verify cached token counts in usage before assuming savings.

Fallback masking

Automatic fallback is great until a degraded provider silently shifts traffic to a pricier one. Meter per-model and graph it. A sudden shift in model mix is a signal, not a win.

Production checklist

• Token distribution logged by feature for 7 days
• Task classes defined at call site, not inferred
• Single MODEL_MAP with cheapest acceptable model per class
• maxTokens set on every call
• Daily token budget enforced with alerting
• Gateway metering verified against provider bills
• A/B comparison of output quality for routed vs unrouted paths

Verce ai sdk cost-aware model routing is not a one-time config; it’s a layer you own. Treat the model map like a pricing table—review it when new models drop, and demote tasks aggressively when a smaller model proves itself.