Next.js API routes vs Edge Functions for LLM streaming

Choosing between next.js api routes vs edge functions llm streaming setups determines where your token stream terminates, how you handle backpressure, and what you pay for idle compute. Both run on Vercel’s platform and integrate with the App Router, but they differ sharply in execution environment, streaming primitives, and operational limits. This comparison breaks down the tradeoffs with concrete code from the Vercel AI SDK and real deployment constraints.

Execution environment and streaming primitives

A standard App Router API route defaults to the Node.js runtime. You opt into Edge by exporting runtime = 'edge' from the route file. The Vercel AI SDK hides most differences behind streamText(), but the underlying stream handling and available APIs diverge.

// app/api/chat/node/route.ts
import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';

export const runtime = 'nodejs';
export const maxDuration = 30;

export async function POST(req: Request) {
  const { messages } = await req.json();
  const result = await streamText({
    model: openai('gpt-4o-mini'),
    messages,
  });
  return result.toDataStreamResponse();
}

// app/api/chat/edge/route.ts
import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';

export const runtime = 'edge';

export async function POST(req: Request) {
  const { messages } = await req.json();
  const result = await streamText({
    model: openai('gpt-4o-mini'),
    messages,
  });
  return result.toDataStreamResponse();
}

Under the hood, both return a ReadableStream wired to the Response. On Node, the stream is backed by the HTTP server’s writable side with full Node event-loop access. On Edge, the stream is a V8 isolate Web ReadableStream with stricter memory and API limits.

Capabilities

Node API routes give you the entire npm ecosystem: fs, child_process, native bindings, and long-running loops. You can buffer large RAG contexts, call a Postgres pool, or run CPU-heavy prompt templating without fear of isolate constraints.

Edge Functions execute in a sandbox with Web Standard APIs only. You get fetch, ReadableStream, crypto.subtle, but no Node built-ins. For LLM streaming, this is usually enough—you are mostly proxying tokens—but any pre-processing that needs pdf-parse or sharp will fail at build time.

If you route through n4n.ai—an OpenAI-compatible gateway covering 240+ models with per-token metering and automatic fallback—the runtime only changes how you forward the stream, not the model logic. The gateway honors client routing directives and forwards provider cache-control hints, so either runtime can act as a thin proxy.

Price and cost model

Vercel bills Node functions by wall-clock invocation duration and allocated memory (GB-seconds). An LLM stream that stays open for 90 seconds while the model generates tokens bills you for the full 90 seconds, even though your function is idle waiting on the upstream socket.

Edge Functions bill by invocations and CPU time, not wall-clock. A streaming proxy that spends 200 ms of CPU shuffling chunks and 90 seconds waiting on the model provider costs only for that 200 ms plus the invocation. For chat apps with long generations, this difference is significant.

There is no free lunch: Edge CPU is metered in microsecond increments and can get expensive if you do heavy transformation per token. But for pass-through streaming, Edge is consistently cheaper.

Latency and throughput

Edge Functions deploy to many regions and cold-start in single-digit milliseconds. The first token from your LLM still depends on the model provider’s location. If you call api.openai.com from an edge node in Frankfurt, the cross-Atlantic hop dominates latency regardless of runtime.

Node functions on Vercel run in a single region (or a few if you configure). Cold starts are higher (100–500 ms typical) but throughput for chunked transfer is comparable. For high-concurrency streaming, Edge scales faster; Node scales per-function concurrency with potential queueing.

Ergonomics

The Vercel AI SDK makes both paths nearly identical, but Edge forces you to avoid Node-specific imports. A common gotcha: using Buffer in Edge throws. Use Uint8Array and TextEncoder instead.

// Edge-safe manual stream transform
export const runtime = 'edge';

export async function POST(req: Request) {
  const upstream = await fetch('https://api.example.com/stream', {
    method: 'POST',
    body: req.body,
  });
  const encoder = new TextEncoder();
  const stream = upstream.body!.pipeThrough(
    new TransformStream({
      transform(chunk, controller) {
        controller.enqueue(encoder.encode(`data: ${chunk}\n\n`));
      },
    })
  );
  return new Response(stream, {
    headers: { 'content-type': 'text/event-stream' },
  });
}

On Node, you could use streaming-response helpers from older Express patterns, but App Router unifies on Web streams.

Ecosystem and tooling

Node routes integrate with Prisma, LangChain, LlamaIndex, and any library that uses process.env or filesystem access. Edge routes work with lightweight clients and Web-only builds of those libraries (e.g., @langchain/core but not full langchain Node bindings).

Local dev with next dev runs both runtimes, but Edge emulation is less faithful than Node. You will catch Edge incompatibilities only at vercel build or runtime.

Limits

The table below summarizes hard constraints that affect LLM streaming deployments on Vercel.

Which to choose

Use Edge Functions when:

• You build a chat UI with the Vercel AI SDK and mostly proxy model output.
• Generations are under 30 seconds of wall-clock (most conversational turns).
• You want lower cost for long streams and global low-latency entry points.
• Your pre-processing is limited to JSON parsing and header manipulation.

Use Node API routes when:

• You run RAG with vector DB clients, file parsing, or server-side aggregation before the LLM call.
• You need longer timeouts for agentic loops or document synthesis.
• You depend on npm packages that assume Node (many LangChain tools, pdf-parse, docx).
• You want to reuse existing Express/Node middleware patterns.

Hybrid pattern: Put auth and lightweight streaming proxy on Edge, then call a Node route (or a background worker) for heavy context assembly. The edge layer can forward provider cache-control hints and client routing directives to a gateway, keeping the hot path cheap.

For most teams shipping a Next.js AI chat integration on Vercel today, start with Edge for the streaming endpoint and move to Node only when you hit the 30-second wall or need a Node-only library. The next.js api routes vs edge functions llm decision is not permanent—the AI SDK’s uniform streamText interface lets you flip the runtime export and re-deploy with minimal changes.