A/B testing GPT-4o vs Llama 3.1 405B with Vercel AI SDK

A/B testing LLMs in production requires more than swapping model names. You need consistent interfaces, measurable outcomes, and a way to route traffic without rewriting your application logic. Vercel AI SDK provides the primitives to do this cleanly. This guide walks through setting up a controlled comparison between GPT-4o and Llama 3.1 405B, from provider configuration through statistical validation.

Step 1: Set up provider clients with a unified interface

Vercel AI SDK’s createOpenAI and createAnthropic factories return compatible LanguageModel instances. For Llama 3.1 405B, you’ll need a provider that serves it — Together AI, Fireworks, or a self-hosted endpoint all expose OpenAI-compatible APIs. Configure each client once at application startup.

// lib/models.ts
import { createOpenAI } from '@ai-sdk/openai';
import { createTogetherAI } from '@ai-sdk/togetherai';

export const gpt4o = createOpenAI({
  apiKey: process.env.OPENAI_API_KEY,
})('gpt-4o');

export const llama405b = createTogetherAI({
  apiKey: process.env.TOGETHER_API_KEY,
})('meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo');

export type ModelId = 'gpt-4o' | 'llama-3.1-405b';

export function getModel(id: ModelId) {
  switch (id) {
    case 'gpt-4o':
      return gpt4o;
    case 'llama-3.1-405b':
      return llama405b;
  }
}

If you’re routing through a gateway like n4n.ai that exposes 240+ models behind one OpenAI-compatible endpoint, you can simplify this to a single client with model selection via the model parameter — but the pattern above keeps provider concerns explicit, which matters when debugging latency or token accounting differences.

Step 2: Build an assignment service with persistent buckets

Random assignment per request destroys statistical power. You need stable buckets tied to a user or session identifier. Implement a deterministic hash-based assignment that survives deployments.

// lib/experiment.ts
import { createHash } from 'crypto';

export type Variant = 'control' | 'treatment';

export interface Assignment {
  variant: Variant;
  modelId: ModelId;
  experimentId: string;
}

const EXPERIMENT_ID = 'gpt4o-vs-llama405b-2024-01';
const CONTROL_MODEL: ModelId = 'gpt-4o';
const TREATMENT_MODEL: ModelId = 'llama-3.1-405b';
const TREATMENT_RATIO = 0.5; // 50/50 split

export function assignVariant(userId: string): Assignment {
  const hash = createHash('sha256')
    .update(`${EXPERIMENT_ID}:${userId}`)
    .digest('hex');
  const bucket = parseInt(hash.slice(0, 8), 16) / 0xffffffff;
  
  const variant: Variant = bucket < TREATMENT_RATIO ? 'treatment' : 'control';
  const modelId = variant === 'treatment' ? TREATMENT_MODEL : CONTROL_MODEL;
  
  return { variant, modelId, experimentId: EXPERIMENT_ID };
}

Store the assignment in your session or user record so repeat requests hit the same variant. This also lets you analyze per-user metrics later.

Step 3: Create a streaming chat endpoint that respects assignment

Your route handler should accept the user’s message history, resolve the assigned model, and stream the response. Use Vercel AI SDK’s streamText for consistent streaming across providers.

// app/api/chat/route.ts
import { streamText } from 'ai';
import { assignVariant } from '@/lib/experiment';
import { getModel } from '@/lib/models';
import { cookies } from 'next/headers';

export async function POST(req: Request) {
  const { messages, userId } = await req.json();
  
  if (!userId) {
    return new Response('userId required', { status: 400 });
  }

  const assignment = assignVariant(userId);
  const model = getModel(assignment.modelId);

  const result = await streamText({
    model,
    messages,
    temperature: 0.7,
    maxTokens: 2048,
    onFinish: async ({ usage, finishReason }) => {
      await logExperimentEvent({
        userId,
        experimentId: assignment.experimentId,
        variant: assignment.variant,
        model: assignment.modelId,
        promptTokens: usage.promptTokens,
        completionTokens: usage.completionTokens,
        finishReason,
        timestamp: new Date().toISOString(),
      });
    },
  });

  return result.toDataStreamResponse({
    headers: {
      'x-experiment-variant': assignment.variant,
      'x-experiment-model': assignment.modelId,
    },
  });
}

The onFinish callback captures token usage per variant — critical for cost analysis. The response headers let the client verify which model handled the request without parsing the stream.

Step 4: Instrument client-side verification

Add a lightweight client hook that reads the experiment headers and exposes them for debugging and analytics.

// hooks/useChatExperiment.ts
import { useChat } from 'ai/react';
import { useEffect, useState } from 'react';

export function useChatExperiment(userId: string) {
  const [variant, setVariant] = useState<'control' | 'treatment' | null>(null);
  const [model, setModel] = useState<string | null>(null);

  const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat({
    api: '/api/chat',
    body: { userId },
    onResponse: (res) => {
      setVariant(res.headers.get('x-experiment-variant') as 'control' | 'treatment');
      setModel(res.headers.get('x-experiment-model'));
    },
  });

  return { messages, input, handleInputChange, handleSubmit, isLoading, variant, model };
}

In your chat UI, render a small badge showing the active variant. This makes manual verification trivial during development and gives support teams a quick diagnostic when users report issues.

Step 5: Define evaluation criteria before you ship

Don’t decide what “better” means after you have data. Specify primary and guardrail metrics upfront. For a general-purpose chat comparison, typical metrics include:

Build an evaluation harness that runs a fixed prompt set against both models. This gives you ground truth before live traffic.

// scripts/evaluate.ts
import { generateText } from 'ai';
import { gpt4o, llama405b } from '@/lib/models';

const EVAL_PROMPTS = [
  { id: 'coding-1', prompt: 'Write a Python function that merges two sorted lists.', expected: 'def merge_sorted...' },
  { id: 'reasoning-1', prompt: 'If all Bloops are Razzies and all Razzies are Lazzies, are all Bloops Lazzies?', expected: 'yes' },
  { id: 'creative-1', prompt: 'Write a haiku about distributed systems.', expected: /haiku structure/ },
];

async function evaluateModel(model: any, name: string) {
  const results = [];
  for (const { id, prompt, expected } of EVAL_PROMPTS) {
    const start = Date.now();
    const { text, usage } = await generateText({ model, prompt, temperature: 0 });
    const latency = Date.now() - start;
    const passed = typeof expected === 'string' 
      ? text.includes(expected) 
      : expected.test(text);
    results.push({ id, passed, latency, tokens: usage.totalTokens, output: text });
  }
  return { name, results };
}

const [gptResults, llamaResults] = await Promise.all([
  evaluateModel(gpt4o, 'gpt-4o'),
  evaluateModel(llama405b, 'llama-3.1-405b'),
]);

console.table(gptResults.results.map(r => ({ task: r.id, pass: r.passed, latency: r.latency, tokens: r.tokens })));
console.table(llamaResults.results.map(r => ({ task: r.id, pass: r.passed, latency: r.latency, tokens: r.tokens })));

Run this weekly. Model behavior drifts; your evaluation set catches regressions before users do.

Step 6: Collect live metrics with structured logging

Your onFinish callback from Step 3 writes to logExperimentEvent. Implement that function to emit structured JSON to your observability stack (Datadog, Honeycomb, Loki, or even a Postgres table).

// lib/telemetry.ts
import { createClient } from '@supabase/supabase-js';

const supabase = createClient(
  process.env.SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_ROLE_KEY!
);

export async function logExperimentEvent(event: ExperimentEvent) {
  const { error } = await supabase.from('experiment_events').insert(event);
  if (error) console.error('Failed to log experiment event:', error);
}

export interface ExperimentEvent {
  userId: string;
  experimentId: string;
  variant: 'control' | 'treatment';
  model: string;
  promptTokens: number;
  completionTokens: number;
  finishReason: string;
  timestamp: string;
}

Add a unique constraint on (experiment_id, user_id) if you want to enforce one assignment per user per experiment at the database level.

Step 7: Analyze results with proper statistics

After collecting sufficient data (minimum 1000 events per variant for binary metrics, more for continuous), run a statistical test. Don’t eyeball percentages.

# analysis/analyze.py
import pandas as pd
from scipy.stats import chi2_contingency, mannwhitneyu
import json

df = pd.read_json('experiment_events.jsonl', lines=True)

# Primary: task success rate (requires user feedback column)
# For this example, assume we have a 'success' boolean from post-chat surveys
contingency = pd.crosstab(df['variant'], df['success'])
chi2, p, dof, expected = chi2_contingency(contingency)
print(f"Task success: chi2={chi2:.3f}, p={p:.4f}")

# Guardrail: latency
control_lat = df[df['variant'] == 'control']['latency_ms']
treatment_lat = df[df['variant'] == 'treatment']['latency_ms']
stat, p_lat = mannwhitneyu(treatment_lat, control_lat, alternative='greater')
print(f"Latency (treatment > control): U={stat:.0f}, p={p_lat:.4f}")

# Guardrail: cost per 1k tokens
df['cost_usd'] = df.apply(lambda r: 
    (r['prompt_tokens'] * 5 + r['completion_tokens'] * 15) / 1_000_000  # GPT-4o pricing
    if r['model'] == 'gpt-4o' 
    else (r['prompt_tokens'] + r['completion_tokens']) * 0.9 / 1_000_000,  # Llama 405B on Together
    axis=1
)
stat, p_cost = mannwhitneyu(
    df[df['variant'] == 'treatment']['cost_usd'],
    df[df['variant'] == 'control']['cost_usd'],
    alternative='less'
)
print(f"Cost (treatment < control): U={stat:.0f}, p={p_cost:.4f}")

Set your significance threshold before the experiment (typically α = 0.05). Apply Bonferroni correction if testing multiple primary hypotheses.

Step 8: Automate rollout or rollback

Wire the analysis to a feature flag. If treatment wins on primary and passes all guardrails, ramp to 100%. If it fails any guardrail, kill it automatically.

// lib/rollout.ts
import { getExperimentResults } from '@/lib/analysis';

export async function evaluateRollout(experimentId: string): Promise<'ramp' | 'hold' | 'rollback'> {
  const results = await getExperimentResults(experimentId);
  
  const primaryPasses = results.primary.pValue < 0.05 && results.primary.lift > 0;
  const latencyPasses = results.guardrails.latency.pValue > 0.05; // treatment not worse
  const costPasses = results.guardrails.cost.pValue < 0.05; // treatment cheaper
  const hallucinationPasses = results.guardrails.hallucination.pValue > 0.05;
  
  if (primaryPasses && latencyPasses && costPasses && hallucinationPasses) {
    return 'ramp';
  }
  if (!latencyPasses || !hallucinationPasses) {
    return 'rollback';
  }
  return 'hold';
}

Schedule this as a daily cron. The first run after reaching statistical power decides the outcome.

Step 9: Verify end-to-end with a smoke test

Before enabling the experiment for real users, run a synthetic verification that exercises the full path: assignment → model call → logging → analysis.

# scripts/smoke-test.sh
#!/bin/bash
set -e

USER_ID="smoke-test-$(date +%s)"
echo "Testing with user: $USER_ID"

# Request 1 - should get consistent assignment
RESPONSE=$(curl -s -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Say hello\"}],\"userId\":\"$USER_ID\"}")

VARIANT=$(echo "$RESPONSE" | grep -o '"x-experiment-variant":"[^"]*' | cut -d'"' -f4)
MODEL=$(echo "$RESPONSE" | grep -o '"x-experiment-model":"[^"]*' | cut -d'"' -f4)

echo "Variant: $VARIANT"
echo "Model: $MODEL"

# Request 2 - same user, must get same variant
RESPONSE2=$(curl -s -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Say hello again\"}],\"userId\":\"$USER_ID\"}")

VARIANT2=$(echo "$RESPONSE2" | grep -o '"x-experiment-variant":"[^"]*' | cut -d'"' -f4)

if [ "$VARIANT" != "$VARIANT2" ]; then
  echo "FAIL: Assignment not stable across requests"
  exit 1
fi

echo "PASS: Assignment stable, model responding
# Verify event logged
sleep 2
EVENT_COUNT=$(curl -s "http://localhost:3000/api/debug/experiment-events?userId=$USER_ID" | jq '.count')
if [ "$EVENT_COUNT" -lt 1 ]; then
  echo "FAIL: No experiment event logged"
  exit 1
fi

echo "PASS: Event logged successfully"

Run this in CI on every deploy. It catches assignment bugs, logging failures, and model configuration drift before they hit production.

Step 10: Document the experiment for future reference

Create a lightweight experiment registry entry that captures the hypothesis, configuration, and outcome. This prevents re-running the same comparison six months later because nobody recorded the result.

# experiments/gpt4o-vs-llama405b-2024-01.md

## Hypothesis
Llama 3.1 405B matches GPT-4o on general chat quality at lower cost and acceptable latency.

## Configuration
- Control: GPT-4o (OpenAI)
- Treatment: Llama 3.1 405B Instruct Turbo (Together AI)
- Split: 50/50
- Assignment: SHA-256(user_id + experiment_id) mod 1
- Minimum sample: 2000 per variant
- Duration cap: 14 days

## Metrics
| Metric | Type | Threshold |
|--------|------|-----------|
| User-rated success | Primary | p < 0.05, lift ≥ 0 |
| p95 latency | Guardrail | Treatment ≤ 1.2x Control |
| Cost/1k tokens | Guardrail | Treatment ≤ Control |
| Hallucination rate (eval) | Guardrail | Treatment ≤ Control |

## Outcome
[Filled in after analysis]
- Decision: [ramp/hold/rollback]
- Primary p-value: 
- Guardrail results:
- Notes:

Commit this file when you launch the experiment. Update the Outcome section when you conclude. Your future self will thank you.

This pattern scales. The same assignment, logging, and analysis infrastructure works for comparing any two models — different providers, different sizes, different fine-tunes. The key is treating the experiment as a first-class engineering artifact: versioned, instrumented, and automatically evaluated.