n4nAI
Open console →
API integrationTutorialVercel Edge Functions LLM Streaming

Streaming LLM responses with Vercel Edge Functions

Learn to build a Vercel Edge Function that streams LLM responses from an OpenAI-compatible API, with runnable code and testing steps.

n4n Team2 min read536 words

Audio narration

Coming soon — every post will get a voice note here.

Streaming LLM responses through Vercel Edge Functions puts inference close to your users and hides network latency behind incremental tokens. This tutorial builds a production-shaped edge proxy that forwards streaming requests to an OpenAI-compatible endpoint, using vercel edge functions llm streaming patterns you can drop into an existing app. You’ll get runnable code, deployment steps, and a curl checkpoint at each stage.

Prerequisites

  • Node.js 18+ and a Vercel account.
  • A Vercel project initialized locally (vercel init or vercel link).
  • An API key for an OpenAI-compatible LLM provider, stored as LLM_API_KEY in .env.local and Vercel env.
  • TypeScript familiarity and basic knowledge of the Web Streams API (ReadableStream, TransformStream).

Why edge instead of Node functions

Vercel’s Node functions run in a microVM with a cold start cost and a single region. Edge Functions execute on V8 isolates distributed across Vercel’s global network. For vercel edge functions llm streaming, that means the TCP connection to the LLM provider terminates near the user, and the first token arrives faster. The trade-off is a CPU time limit and no Node built-ins, but streaming I/O is fully supported.

Project setup

Create api/stream.ts. Vercel maps files under api/ to routes; the runtime: 'edge' export switches the runtime.

// api/stream.ts
export const config = { runtime: 'edge' };

export default async function handler(_req: Request): Promise<Response> {
  return new Response('ok');
}

Run vercel dev and hit it:

curl http://localhost:3000/api/stream
# ok

Proxying the stream

The handler reads a JSON prompt, calls the provider with stream: true, and returns the upstream body. Edge fetch yields a streaming Response; we forward its body directly.

// api/stream.ts
export const config = { runtime: 'edge' };

const LLM_ENDPOINT = 'https://api.openai.com/v1/chat/completions';

export default async function handler(req: Request): Promise<Response> {
  if (req.method !== 'POST') return new Response('Method not allowed', { status: 405 });

  const { prompt } = await req.json().catch(() => ({}));
  if (!prompt) return new Response('Missing prompt', { status: 400 });

  const upstream = await fetch(LLM_ENDPOINT, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${process.env.LLM_API_KEY}`,
    },
    body: JSON.stringify({
      model: 'gpt-3.5-turbo',
      messages: [{ role: 'user', content: prompt }],
      stream: true,
    }),
  });

  return new Response(upstream.body, {
    status: upstream.status,
    headers: {
      'Content-Type': 'text/event-stream; charset=utf-8',
      'Cache-Control': 'no-cache',
    },
  });
}

This is the minimal vercel edge functions llm streaming proxy. The provider’s Server-Sent Events (SSE) flow untouched to the browser.

Test:

curl -N -X POST http://localhost:3000/api/stream \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Explain edge runtimes in one sentence."}'

Expected output (truncated):

data: {"choices":[{"delta":{"role":"assistant"}}]}
data: {"choices":[{"delta":{"content":"Edge"}}]}
data: {"choices":[{"delta":{"content":" runtimes"}}]}
data: [DONE]

Supporting chat history

Swap the single prompt for a messages array to match standard chat clients:

const { messages } = await req.json().catch(() => ({}));
if (!Array.isArray(messages) || messages.length === 0) {
  return new Response('Missing messages', { status: 400 });
}
// use `messages` in the body instead of constructing from prompt

This keeps the vercel edge functions llm streaming proxy compatible with multi-turn UIs.

Stripping SSE framing

Most chat UIs want raw text, not SSE. Use a TransformStream to parse lines and emit only content deltas.

const decoder = new TextDecoder();
const encoder = new TextEncoder();

const transform = new TransformStream({
  transform(chunk, controller) {
    const text = decoder.decode(chunk, { stream: true });
    for (const line of text.split('\n')) {
      const trimmed = line.trim();
      if (!trimmed.startsWith('data:')) continue;
      const payload = trimmed.slice(5).trim();
      if (payload === '[DONE]') continue;
      try {
        const json = JSON.parse(payload);
        const delta = json.choices?.[0]?.delta?.content;
        if (delta) controller.enqueue(encoder.encode(delta));
      } catch {
        // ignore keep-alive or partial JSON
      }
    }
  },
});

return new Response(upstream.body?.pipeThrough(transform), {
  headers: { 'Content-Type': 'text/plain; charset=utf-8' },
});

Now the client gets Edge runtimes... as a continuous byte stream. This pattern is the backbone of vercel edge functions llm streaming integrations where you own the protocol.

Browser consumption

A minimal client reader:

const res = await fetch('/api/stream', {
  method: 'POST',
  body: JSON.stringify({ messages: [{ role: 'user', content: 'Hello' }] }),
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  document.body.append(decoder.decode(value));
}

In React, store accumulated text in state and render it.

Error handling and fallbacks

If the upstream returns 429 or 500, upstream.ok is false. Return its body so the client sees the error. For resilience, point at a gateway that aggregates providers. n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and applies automatic fallback when a provider is rate-limited or degraded. The edge code stays identical; swap LLM_ENDPOINT and key. It also forwards provider cache-control hints, so edge caching behaves.

const LLM_ENDPOINT = 'https://api.n4n.ai/v1/chat/completions';

Debugging tips

Run curl -N -i http://localhost:3000/api/stream to inspect status line and headers. In vercel dev, uncaught exceptions print to the terminal; wrap the fetch in try/catch to return a 502 with a clear message. Streaming transforms that throw inside transform will abort the response—log partial chunks to avoid silent failures.

Production checklist

  • Store secrets with vercel env add LLM_API_KEY production.
  • Set LLM_ENDPOINT and LLM_MODEL via env to avoid code changes per environment.
  • Add CORS if the frontend sits on a different domain: 
    headers: { 'Access-Control-Allow-Origin': 'https://your.app' }
  • Edge Functions have a max duration (verify current limit in Vercel docs); streaming connections count against it.
  • Log with console.error — edge logs ship to Vercel’s dashboard.

Full handler

export const config = { runtime: 'edge' };

const LLM_ENDPOINT = process.env.LLM_ENDPOINT ?? 'https://api.openai.com/v1/chat/completions';

export default async function handler(req: Request): Promise<Response> {
  if (req.method !== 'POST') return new Response('Method not allowed', { status: 405 });
  const { messages } = await req.json().catch(() => ({}));
  if (!Array.isArray(messages) || messages.length === 0) {
    return new Response('Missing messages', { status: 400 });
  }

  const upstream = await fetch(LLM_ENDPOINT, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${process.env.LLM_API_KEY}`,
    },
    body: JSON.stringify({
      model: process.env.LLM_MODEL ?? 'gpt-3.5-turbo',
      messages,
      stream: true,
    }),
  });

  if (!upstream.ok || !upstream.body) {
    return new Response(upstream.body, { status: upstream.status });
  }

  const decoder = new TextDecoder();
  const encoder = new TextEncoder();
  const transform = new TransformStream({
    transform(chunk, controller) {
      const text = decoder.decode(chunk, { stream: true });
      for (const line of text.split('\n')) {
        const t = line.trim();
        if (!t.startsWith('data:')) continue;
        const p = t.slice(5).trim();
        if (p === '[DONE]') return;
        try {
          const json = JSON.parse(p);
          const delta = json.choices?.[0]?.delta?.content;
          if (delta) controller.enqueue(encoder.encode(delta));
        } catch {}
      }
    },
  });

  return new Response(upstream.body.pipeThrough(transform), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8', 'Cache-Control': 'no-cache' },
  });
}

Deploy with vercel --prod. You now run a global vercel edge functions llm streaming endpoint that proxies any OpenAI-compatible model.

Tagsverceledge-functionsstreamingllm-api

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationDeploying an OpenAI-compatible proxy on Vercel Edge FunctionsPractical step-by-step tutorial for building and deploying an OpenAI-compatible LLM proxy on Vercel Edge Functions with streaming, CORS, and fallback.API integrationHandling CORS for LLM streaming on Vercel Edge FunctionsSet up CORS for LLM streaming on Vercel Edge Functions with correct preflight and response headers, using a proxy pattern that works in production.API integrationReducing time-to-first-token with Vercel Edge FunctionsPractical guide to cutting vercel edge functions time to first token for LLM streaming: edge proxy patterns, code, pitfalls, and latency tradeoffs.API integrationVercel Edge Functions and n4n.ai: a streaming chat walkthroughBuild a streaming chat API on Vercel Edge Functions using the n4n.ai OpenAI-compatible gateway. Step-by-step tutorial with runnable TypeScript code.API integrationEdge streaming with the Web Streams API on VercelLearn how to implement vercel web streams api edge streaming for LLM responses on Vercel Edge Functions with runnable code and verification steps.API integrationVercel Edge Config for managing LLM API routing rulesLearn how to use Vercel Edge Config to manage LLM API routing rules at the edge, with runnable code for dynamic provider failover and model mapping.

All vercel edge functions llm streaming posts →