Vercel AI SDK setup: Llama 3.1 70B via n4n.ai

The Vercel AI SDK gives you a clean abstraction over LLM providers, but wiring it to a gateway like n4n.ai requires a few deliberate choices: base URL configuration, model naming, and handling the OpenAI-compatible response format. This tutorial walks through a complete, production-ready setup — streaming, tool calling, and structured error handling included.

Prerequisites

• Node.js 18.17 or later (required for fetch and ReadableStream globals)
• An n4n.ai API key — get one at n4n.ai
• A Next.js 14+ project (App Router) or a standalone Node/TypeScript project
• Basic familiarity with the Vercel AI SDK streamText and generateText APIs

If you’re starting fresh:

npx create-next-app@latest ai-sdk-demo --typescript --tailwind --eslint --app --src-dir --import-alias "@/*"
cd ai-sdk-demo
npm install ai @ai-sdk/openai zod

The @ai-sdk/openai package works because n4n.ai exposes an OpenAI-compatible endpoint. No custom provider code needed.

Configure the gateway client

Create a singleton client that points at n4n.ai. Keep it in a dedicated module so you can swap configuration without touching route handlers.

// lib/gateway.ts
import { createOpenAI } from '@ai-sdk/openai';

export const gateway = createOpenAI({
  baseURL: 'https://api.n4n.ai/v1',
  apiKey: process.env.N4N_API_KEY,
  // Optional: forward client routing directives
  // headers: { 'x-n4n-routing': 'latency-optimized' },
});

Add your key to .env.local:

N4N_API_KEY=n4n_sk_yourkeyhere

Restart the dev server after adding the env var.

First request: non-streaming generation

Verify the pipeline works with a simple generateText call. Create a test route:

// app/api/test/route.ts
import { generateText } from 'ai';
import { gateway } from '@/lib/gateway';

export async function GET() {
  const { text } = await generateText({
    model: gateway('meta-llama/llama-3.1-70b-instruct'),
    prompt: 'Say "pong" and nothing else.',
    temperature: 0,
    maxTokens: 10,
  });

  return Response.json({ text });
}

Hit http://localhost:3000/api/test. Expected output:

{"text":"pong"}

If you see a 401, check the API key. If you see a 404 on the model, verify the exact model slug — n4n.ai uses provider-style IDs like meta-llama/llama-3.1-70b-instruct.

Streaming responses with streamText

Streaming is where the AI SDK shines. Replace the test route with a chat endpoint that streams tokens to the client.

// app/api/chat/route.ts
import { streamText } from 'ai';
import { gateway } from '@/lib/gateway';

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: gateway('meta-llama/llama-3.1-70b-instruct'),
    messages,
    temperature: 0.7,
    maxTokens: 512,
  });

  return result.toDataStreamResponse();
}

The toDataStreamResponse() helper returns a ReadableStream formatted for the AI SDK’s useChat hook. No manual chunk parsing.

Build a minimal chat UI

Use the useChat hook from ai/react. This handles optimistic updates, streaming render, and retry logic.

// app/page.tsx
'use client';

import { useChat } from 'ai/react';

export default function Chat() {
  const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat({
    api: '/api/chat',
  });

  return (
    <main className="flex min-h-screen flex-col items-center p-8 gap-4">
      <div className="w-full max-w-2xl space-y-4">
        {messages.map((m) => (
          <div key={m.id} className={`whitespace-pre-wrap ${m.role === 'user' ? 'text-right' : ''}`}>
            <strong>{m.role === 'user' ? 'You' : 'Llama 3.1 70B'}:</strong> {m.content}
          </div>
        ))}
      </div>

      <form onSubmit={handleSubmit} className="w-full max-w-2xl flex gap-2">
        <input
          value={input}
          onChange={handleInputChange}
          placeholder="Ask anything..."
          className="flex-1 border rounded px-3 py-2"
          disabled={isLoading}
        />
        <button type="submit" disabled={isLoading || !input.trim()} className="px-4 py-2 bg-blue-600 text-white rounded disabled:opacity-50">
          Send
        </button>
      </form>
    </main>
  );
}

Run npm run dev, open http://localhost:3000, and send a message. You should see tokens appear incrementally.

Tool calling: give the model a calculator

Llama 3.1 70B supports function calling. Define a tool with Zod schemas and pass it to streamText.

// app/api/chat/route.ts
import { streamText, tool } from 'ai';
import { gateway } from '@/lib/gateway';
import { z } from 'zod';

const calculator = tool({
  parameters: z.object({
    expression: z.string().describe('Expression to evaluate, e.g. "(2 + 3) * 4"'),
  }),
  execute: async ({ expression }) => {
    // Safe eval for demo — use a real expression parser in production
    try {
      const result = Function(`"use strict"; return (${expression})`)();
      return { result };
    } catch {
      return { error: 'Invalid expression' };
    }
  },
});

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: gateway('meta-llama/llama-3.1-70b-instruct'),
    messages,
    tools: { calculator },
    temperature: 0.3,
    maxTokens: 1024,
  });

  return result.toDataStreamResponse();
}

Test it: ask “What’s (17 * 23) + 42?” The model calls the tool, receives the result, and composes the answer. Expected streamed output includes a tool call chunk, then the final text.

Structured output with generateObject

For non-chat use cases — extraction, classification, JSON APIs — use generateObject with a Zod schema.

// app/api/extract/route.ts
import { generateObject } from 'ai';
import { gateway } from '@/lib/gateway';
import { z } from 'zod';

const schema = z.object({
  name: z.string(),
  email: z.string().email(),
  company: z.string().optional(),
  intent: z.enum(['sales', 'support', 'partnership', 'other']),
});

export async function POST(req: Request) {
  const { text } = await req.json();

  const { object } = await generateObject({
    model: gateway('meta-llama/llama-3.1-70b-instruct'),
    schema,
    prompt: `Extract structured data from this inquiry:\n\n${text}`,
    temperature: 0,
  });

  return Response.json(object);
}

POST { "text": "Hi, I'm Jane from Acme Corp (jane@acme.com). We need enterprise pricing." } — you get typed JSON back.

Error handling and retries

The AI SDK surfaces provider errors as AIError subclasses. Wrap route handlers to return consistent error shapes.

// lib/with-error-handling.ts
import { AIError } from 'ai';
import { NextResponse } from 'next/server';

export function withErrorHandling(handler: (req: Request) => Promise<Response>) {
  return async (req: Request) => {
    try {
      return await handler(req);
    } catch (err) {
      if (err instanceof AIError) {
        return NextResponse.json(
          { error: err.message, code: err.code },
          { status: err.statusCode ?? 500 }
        );
      }
      console.error('[chat]', err);
      return NextResponse.json({ error: 'Internal server error' }, { status: 500 });
    }
  };
}

Apply it:

// app/api/chat/route.ts
import { withErrorHandling } from '@/lib/with-error-handling';
// ... existing imports

export const POST = withErrorHandling(async (req: Request) => {
  // ... handler body
});

Common codes you’ll see: RATE_LIMITED (429), MODEL_UNAVAILABLE (503), INVALID_API_KEY (401). The gateway returns standard OpenAI error shapes, so the SDK maps them correctly.

Provider routing directives

n4n.ai honors client-side routing hints via headers. Use them when you have latency or cost requirements.

// lib/gateway.ts
import { createOpenAI } from '@ai-sdk/openai';

export const gateway = createOpenAI({
  baseURL: 'https://api.n4n.ai/v1',
  apiKey: process.env.N4N_API_KEY,
  headers: {
    // Options: 'latency-optimized', 'cost-optimized', 'quality-optimized'
    'x-n4n-routing': 'latency-optimized',
  },
});

You can also override per-request by passing headers to streamText or generateText:

const result = streamText({
  model: gateway('meta-llama/llama-3.1-70b-instruct'),
  messages,
  headers: { 'x-n4n-routing': 'cost-optimized' },
  // ...
});

Observability: token usage and latency

The streamText result includes a usage promise that resolves after the stream completes. Log it for cost tracking.

const result = streamText({
  model: gateway('meta-llama/llama-3.1-70b-instruct'),
  messages,
  onFinish: async ({ usage, finishReason, response }) => {
    console.log('[chat] usage', {
      promptTokens: usage.promptTokens,
      completionTokens: usage.completionTokens,
      totalTokens: usage.totalTokens,
      finishReason,
      latencyMs: response.headers.get('x-n4n-latency-ms'),
    });
  },
});

The x-n4n-latency-ms header (when present) reports gateway-side latency — useful for SLO dashboards.

Production checklist

Before deploying:

1. Rate limiting — Add middleware (e.g., @vercel/rate-limiter) on /api/chat to prevent abuse.
2. Auth — Gate the chat endpoint behind your auth system (NextAuth, Clerk, etc.).
3. Model pinning — Lock to a specific model version if you need reproducibility: meta-llama/llama-3.1-70b-instruct@sha256:... (check n4n.ai docs for versioned IDs).
4. Timeouts — Set maxTokens and consider abortSignal integration for long-running streams.
5. Fallback — The gateway handles provider failover automatically, but you can implement application-level fallback by catching MODEL_UNAVAILABLE and retrying with a different model slug.

Deploy to Vercel

vercel env add N4N_API_KEY
vercel --prod

The edge runtime works, but Node.js runtime is recommended for streamText due to larger response buffers and better stream handling. Add to your route:

export const runtime = 'nodejs';

What’s next

• RAG — Pipe retrieval results into messages as system context before calling streamText.
• Multi-step agents — Use maxSteps in streamText to enable autonomous tool loops.
• Eval — Capture onFinish payloads to build a golden dataset for prompt regression testing.

The Vercel AI SDK + n4n.ai combination gives you a provider-agnostic layer with production-grade streaming, tooling, and observability — all in ~100 lines of route code.