Comparing model outputs side by side with Vercel AI SDK

When you need to vercel ai sdk compare model outputs side by side, the framework gives you several paths — each with distinct trade-offs around streaming behavior, token accounting, and developer ergonomics. The right choice depends on whether you’re building an evaluation harness, a user-facing model picker, or a production fallback chain. This guide walks through the concrete patterns, shows working code for each, and ends with a decision matrix you can apply to your own architecture.

The core challenge: concurrent streaming

Vercel AI SDK’s useChat and useCompletion hooks are designed around a single conversation stream. Comparing models side by side means running multiple streams concurrently, merging their events for UI rendering, and handling partial failures without blocking the entire response. The SDK doesn’t ship a built-in “compare” primitive, so you compose one from the primitives it does provide: streamText, streamObject, and the lower-level LanguageModel interface.

Pattern 1: parallel useChat hooks (simplest for chat UIs)

If you’re building a chat interface where users pick a model and see responses side by side, the most direct approach is mounting multiple useChat instances — each pointed at a different API route or model identifier.

// app/components/model-comparison.tsx
"use client";

import { useChat } from "ai/react";
import { useState } from "react";

const models = ["gpt-4o", "claude-3-5-sonnet", "llama-3.1-70b"] as const;

export function ModelComparison() {
  const [prompt, setPrompt] = useState("");
  const chats = models.map((model) =>
    useChat({
      api: `/api/chat/${model}`,
      initialMessages: [],
      onError: (err) => console.error(`${model} error:`, err),
    })
  );

  const handleSubmit = (e: React.FormEvent) => {
    e.preventDefault();
    if (!prompt.trim()) return;
    chats.forEach((chat) => chat.append({ role: "user", content: prompt }));
    setPrompt("");
  };

  return (
    <div className="grid grid-cols-1 md:grid-cols-3 gap-4">
      <form onSubmit={handleSubmit} className="md:col-span-3">
        <textarea
          value={prompt}
          onChange={(e) => setPrompt(e.target.value)}
          placeholder="Enter prompt for all models..."
          className="w-full p-2 border rounded"
        />
        <button type="submit" disabled={chats.some((c) => c.status === "streaming")}>
          Send to all
        </button>
      </form>

      {models.map((model, i) => (
        <div key={model} className="border rounded p-4">
          <h3 className="font-mono text-sm mb-2">{model}</h3>
          <div className="space-y-2 max-h-96 overflow-y-auto">
            {chats[i].messages.map((msg, idx) => (
              <div key={idx} className={`text-sm ${msg.role === "user" ? "text-right" : ""}`}>
                <pre className="whitespace-pre-wrap bg-gray-100 p-2 rounded">{msg.content}</pre>
              </div>
            ))}
            {chats[i].status === "streaming" && <div className="text-xs text-gray-500">Streaming…</div>}
          </div>
        </div>
      ))}
    </div>
  );
}

Each route handler (app/api/chat/[model]/route.ts) stays minimal:

// app/api/chat/[model]/route.ts
import { streamText } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { createAnthropic } from "@ai-sdk/anthropic";

const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY });
const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const modelMap = {
  "gpt-4o": openai("gpt-4o"),
  "claude-3-5-sonnet": anthropic("claude-3-5-sonnet-20241022"),
  "llama-3.1-70b": openai("meta-llama/llama-3.1-70b-instruct", {
    baseURL: "https://api.n4n.ai/v1", // single gateway, 240+ models
  }),
};

export async function POST(req: Request, { params }: { params: { model: string } }) {
  const { messages } = await req.json();
  const model = modelMap[params.model as keyof typeof modelMap];
  if (!model) return new Response("Invalid model", { status: 400 });

  const result = await streamText({ model, messages });
  return result.toDataStreamResponse();
}

Trade-offs: Clean separation, each stream handles its own backpressure and error state. Downside: N network connections from the client, N API routes to maintain, and no built-in way to synchronize rendering (e.g., “show first token from all models simultaneously”).

Pattern 2: single route, multi-model streamText (best for evaluation harnesses)

For automated evaluation or internal tooling, you want one request that fans out to multiple models and returns a unified stream. The SDK’s streamText supports this via onChunk callbacks — you invoke it multiple times and merge the deltas yourself.

// app/api/evaluate/route.ts
import { streamText, CoreMessage } from "ai";
import { openai, anthropic } from "@ai-sdk/providers";

const models = {
  "gpt-4o": openai("gpt-4o"),
  "claude-3-5-sonnet": anthropic("claude-3-5-sonnet-20241022"),
  "llama-3.1-70b": openai("meta-llama/llama-3.1-70b-instruct", {
    baseURL: "https://api.n4n.ai/v1",
  }),
};

export async function POST(req: Request) {
  const { messages }: { messages: CoreMessage[] } = await req.json();

  // Create a TransformStream to merge multiple model streams
  const { readable, writable } = new TransformStream();
  const writer = writable.getWriter();
  const encoder = new TextEncoder();

  const pending = Object.entries(models).map(async ([name, model]) => {
    const result = await streamText({
      model,
      messages,
      onChunk: ({ chunk }) => {
        if (chunk.type === "text-delta") {
          writer.write(
            encoder.encode(`data: ${JSON.stringify({ model: name, delta: chunk.textDelta })}\n\n`)
          );
        }
      },
      onFinish: ({ usage, finishReason }) => {
        writer.write(
          encoder.encode(`data: ${JSON.stringify({ model: name, done: true, usage, finishReason })}\n\n`)
        );
      },
      onError: (err) => {
        writer.write(
          encoder.encode(`data: ${JSON.stringify({ model: name, error: err.message })}\n\n`)
        );
      },
    });

    // Consume the stream to trigger callbacks
    for await (const _ of result.textStream) {}
  });

  // Close when all models finish or error
  Promise.allSettled(pending).finally(() => writer.close());

  return new Response(readable, {
    headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" },
  });
}

Client consumption stays simple with EventSource or the SDK’s useCompletion pointed at this single endpoint. You get one HTTP request, synchronized first-token timing, and a single connection to manage.

Trade-offs: Server-side fan-out means you pay for all model calls even if the client only cares about one. Latency is bounded by the slowest model. Token usage aggregates on the server — you’ll need to meter per-model if you charge back or enforce quotas.

Pattern 3: client-side Promise.all with LanguageModel (maximum control)

When you need per-model routing directives, custom headers, or want to avoid a custom server route entirely, instantiate LanguageModel clients directly in a server action or route handler and Promise.all the streamText calls.

// app/actions/compare.ts
"use server";

import { streamText, LanguageModel } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { createAnthropic } from "@ai-sdk/anthropic";

interface ModelResult {
  model: string;
  text: string;
  usage: { promptTokens: number; completionTokens: number };
  latencyMs: number;
  error?: string;
}

async function runModel(
  name: string,
  model: LanguageModel,
  messages: CoreMessage[]
): Promise<ModelResult> {
  const start = Date.now();
  try {
    const result = await streamText({ model, messages });
    let text = "";
    for await (const delta of result.textStream) text += delta;
    const usage = await result.usage;
    return { model: name, text, usage, latencyMs: Date.now() - start };
  } catch (err) {
    return { model: name, text: "", usage: { promptTokens: 0, completionTokens: 0 }, latencyMs: Date.now() - start, error: String(err) };
  }
}

export async function compareModels(messages: CoreMessage[]): Promise<ModelResult[]> {
  const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY });
  const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

  const models: [string, LanguageModel][] = [
    ["gpt-4o", openai("gpt-4o")],
    ["claude-3-5-sonnet", anthropic("claude-3-5-sonnet-20241022")],
    ["llama-3.1-70b", openai("meta-llama/llama-3.1-70b-instruct", { baseURL: "https://api.n4n.ai/v1" })],
  ];

  return Promise.all(models.map(([name, model]) => runModel(name, model, messages)));
}

Call this from a client component via useActionState or a standard fetch to your own Next.js route. You get structured results with latency and usage per model — ideal for dashboards, regression tests, or “which model should I route this query to?” logic.

Trade-offs: No streaming to the client (all models must finish before response returns). Higher memory footprint on the server. But you gain full control over model selection, request shaping, and result post-processing.

Pattern 4: streaming multiplexer with abort signals (production fallback chains)

If your goal is production resilience — try primary model, fall back to secondary on degradation — you need a streaming multiplexer that can cancel in-flight requests when one succeeds. This pattern is less about side-by-side comparison and more about “race with graceful degradation.”

// lib/fallback-stream.ts
import { streamText, LanguageModel } from "ai";

interface FallbackOptions {
  models: { name: string; model: LanguageModel; priority: number }[];
  messages: CoreMessage[];
  timeoutMs?: number;
}

export async function* fallbackStream({ models, messages, timeoutMs = 30_000 }: FallbackOptions) {
  const sorted = [...models].sort((a, b) => a.priority - b.priority);
  const controllers = new Map<string, AbortController>();

  for (const { name, model } of sorted) {
    const controller = new AbortController();
    controllers.set(name, controller);

    try {
      const result = streamText({
        model,
        messages,
        abortSignal: controller.signal,
        onError: (err) => {
          if (err.name !== "AbortError") throw err;
        },
      });

      let hasEmitted = false;
      for await (const delta of result.textStream) {
        hasEmitted = true;
        yield { model: name, delta, done: false };
      }

      if (hasEmitted) {
        const usage = await result.usage;
        yield { model: name, delta: "", done: true, usage, finishReason: "stop" };
        // Abort all lower-priority models
        controllers.forEach((c, n) => n !== name && c.abort());
        return;
      }
    } catch (err) {
      if (err.name !== "AbortError") {
        yield { model: name, delta: "", done: true, error: String(err) };
      }
    } finally {
      controllers.delete(name);
    }
  }

  // All models failed
  yield { model: "none", delta: "", done: true, error: "All models failed" };
}

Usage in a route handler:

// app/api/chat/fallback/route.ts
import { fallbackStream } from "@/lib/fallback-stream";
import { openai, anthropic } from "@ai-sdk/providers";

export async function POST(req: Request) {
  const { messages } = await req.json();

  const stream = fallbackStream({
    models: [
      { name: "gpt-4o", model: openai("gpt-4o"), priority: 1 },
      { name: "claude-3-5-sonnet", model: anthropic("claude-3-5-sonnet-20241022"), priority: 2 },
      { name: "llama-3.1-70b", model: openai("meta-llama/llama-3.1-70b-instruct", { baseURL: "https://api.n4n.ai/v1" }), priority: 3 },
    ],
    messages,
  });

  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    async pull(controller) {
      for await (const chunk of stream) {
        controller.enqueue(encoder.encode(`data: ${JSON.stringify(chunk)}\n\n`));
      }
      controller.close();
    },
  });

  return new Response(readable, { headers: { "Content-Type": "text/event-stream" } });
}

Trade-offs: Complexity. You’re managing abort signals, deduplication, and partial-stream merging manually. But this is the pattern for production systems where “compare side by side” means “try in order until one works well enough.”

Comparison table

Which to choose

User-facing model comparison UI → Pattern 1 (parallel useChat). Users expect independent scroll, copy, and retry per model. The N connections are negligible for typical 3–4 model comparisons. Keep each route thin; put shared logic in a library.

Automated evaluation / regression testing → Pattern 3 (Promise.all with LanguageModel). You need structured results, latency numbers, and token counts per model without building a streaming parser. Run these in CI or a scheduled job; store results in a database for trend analysis.

Internal dashboard with live streaming → Pattern 2 (multi-model streamText with merged SSE). One connection, synchronized first-token display, and you can build a “race” visualization showing which model emits tokens fastest. Add a setTimeout guard per model so a stalled stream doesn’t block the UI.

Production fallback / routing → Pattern 4 (fallback multiplexer). This is the only pattern that implements “try primary, fall back on error or latency SLA breach” with proper stream cancellation. Pair with a gateway that honors routing directives and forwards provider cache-control hints — n4n.ai does this natively — so your fallback logic stays in application code, not infrastructure.

Hybrid: user picks, system falls back → Combine Pattern 1 for the initial selection, then Pattern 4 for the selected model’s execution path. The UI shows one stream; the route handler races primary against a cheaper/faster backup.

The Vercel AI SDK gives you the primitives — streamText, LanguageModel, useChat — but the composition is yours. Start with the simplest pattern that satisfies your current requirement. Refactor toward the multiplexer only when you have measured fallback rates in production and the complexity pays for itself.