The Vercel AI SDK’s streamText and useChat primitives work well on Node.js, but the edge runtime introduces constraints that break naive implementations. Cold starts, response buffering, and provider failures all behave differently when your code executes in a V8 isolate instead of a long-running process. This guide walks through patterns that keep streaming reliable, observable, and cost-aware on edge platforms.
Why edge middleware changes the game
On a traditional Node server, you hold a persistent connection to your LLM provider. On the edge, each request spins up a fresh isolate with a 50–150 ms cold start. You cannot rely on connection pooling, in-memory caches, or background jobs. The request must complete within the platform’s CPU time limit (typically 30–50 seconds on Vercel) and the response must stream without buffering the entire payload.
Middleware sits between your route handler and the provider. It handles retries, fallbacks, token counting, and header normalization before the stream reaches the client. Done right, the client sees a single text/event-stream response regardless of how many provider hops occurred.
Minimal streaming route with middleware
Start with a route that delegates to a middleware wrapper. The handler stays thin; all provider logic lives in the middleware.
// app/api/chat/route.ts
import { streamText } from 'ai';
import { createMiddleware } from '@/lib/edge-middleware';
export const runtime = 'edge';
export async function POST(req: Request) {
const { messages } = await req.json();
const middleware = createMiddleware({
model: 'openai/gpt-4o-mini',
fallbackModels: ['anthropic/claude-3-haiku', 'meta-llama/llama-3.1-70b'],
maxTokens: 4000,
temperature: 0.7,
});
return middleware.streamText({ messages });
}The middleware factory returns an object with a streamText method that matches the AI SDK signature. This keeps your route handlers portable — swap the middleware implementation without touching the route.
Building the middleware layer
The middleware wraps the provider call, handles fallbacks, and injects usage headers. Here’s a production-ready skeleton:
// lib/edge-middleware.ts
import { streamText, CoreMessage, LanguageModel } from 'ai';
import { createOpenAI } from '@ai-sdk/openai';
import { createAnthropic } from '@ai-sdk/anthropic';
interface MiddlewareConfig {
model: string;
fallbackModels: string[];
maxTokens: number;
temperature: number;
}
interface ProviderClient {
modelId: string;
client: LanguageModel;
}
function parseModelId(modelId: string): ProviderClient {
const [provider, ...modelParts] = modelId.split('/');
const model = modelParts.join('/');
switch (provider) {
case 'openai':
return { modelId, client: createOpenAI({ apiKey: process.env.OPENAI_API_KEY })(model) };
case 'anthropic':
return { modelId, client: createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY })(model) };
default:
throw new Error(`Unknown provider: ${provider}`);
}
}
export function createMiddleware(config: MiddlewareConfig) {
const primary = parseModelId(config.model);
const fallbacks = config.fallbackModels.map(parseModelId);
async function* streamWithFallback(messages: CoreMessage[]) {
const models = [primary, ...fallbacks];
let lastError: Error | null = null;
for (const { modelId, client } of models) {
try {
const result = await streamText({
model: client,
messages,
maxTokens: config.maxTokens,
temperature: config.temperature,
});
// Forward provider headers for cache control and usage
const headers = new Headers();
if (result.responseHeaders) {
Object.entries(result.responseHeaders).forEach(([k, v]) => {
headers.set(k, v);
});
}
// Inject our own usage metadata
headers.set('x-model-used', modelId);
headers.set('x-fallback-count', String(models.indexOf({ modelId, client })));
yield { stream: result.textStream, headers };
return; // Success — exit the fallback loop
} catch (err) {
lastError = err as Error;
// Log and continue to next fallback
console.warn(`Model ${modelId} failed:`, err);
continue;
}
}
throw lastError ?? new Error('All models exhausted');
}
return {
async streamText({ messages }: { messages: CoreMessage[] }) {
const generator = streamWithFallback(messages);
const { value: first } = await generator.next();
if (!first) {
throw new Error('No model responded');
}
const { stream, headers } = first;
// Consume remaining fallbacks if first yields done (shouldn't happen)
for await (const _ of generator) {}
return new Response(stream, {
headers: {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache, no-transform',
'Connection': 'keep-alive',
...Object.fromEntries(headers.entries()),
},
});
},
};
}Key points in this implementation:
- Provider abstraction —
parseModelIdmaps a unifiedprovider/modelstring to the correct AI SDK client. Add new providers without touching route handlers. - Ordered fallback — The loop tries each model sequentially. On rate limits (429), timeouts, or 5xx errors, it moves to the next model automatically.
- Header forwarding — Provider response headers like
x-ratelimit-remaining,anthropic-ratelimit-tokens, or cache hints pass through to the client. Your frontend can render quota warnings. - Model attribution — The
x-model-usedheader tells you which model actually served the request. Critical for debugging and cost allocation.
Handling streaming edge cases
Partial responses on fallback
If the primary model streams 200 tokens then fails, you have a choice: restart from scratch on the fallback, or stitch the partial response. Restarting is simpler and safer — the client receives a clean stream from the fallback model. The tradeoff is duplicated tokens (you pay twice for the prefix). For most chat use cases, the cost is negligible compared to the complexity of stitching.
// In streamWithFallback, the try block wraps the entire streamText call.
// If it throws mid-stream, the catch block triggers and we retry the full
// request on the next model. No partial data leaks to the client.Timeout enforcement
Edge runtimes enforce a hard CPU limit. Wrap the stream in a timeout that aborts cleanly:
function withTimeout<T>(promise: Promise<T>, ms: number): Promise<T> {
return Promise.race([
promise,
new Promise<never>((_, reject) =>
setTimeout(() => reject(new Error('Edge timeout')), ms)
),
]);
}
// Usage in streamWithFallback:
const result = await withTimeout(
streamText({ model: client, messages, maxTokens: config.maxTokens }),
25000 // Leave headroom before platform limit
);Backpressure and buffering
The AI SDK’s textStream is an async iterable that respects backpressure. Do not collect it into a string or array — that buffers the entire response in memory and defeats streaming. Pipe directly to the Response constructor as shown above.
If you need to transform the stream (e.g., inject citations, filter PII), use a TransformStream:
const transform = new TransformStream({
async transform(chunk, controller) {
// chunk is a string token from the model
const sanitized = sanitizePII(chunk);
controller.enqueue(sanitized);
},
});
return new Response(stream.pipeThrough(transform), { headers });Token metering and cost observability
You cannot manage what you don’t measure. The middleware should emit usage events to your observability stack. Here’s a lightweight approach using structured logs:
// lib/usage.ts
interface UsageEvent {
timestamp: string;
model: string;
promptTokens: number;
completionTokens: number;
totalTokens: number;
requestId: string;
userId?: string;
fallbackCount: number;
}
export function logUsage(event: UsageEvent) {
// Structured JSON for your log aggregator (Datadog, Loki, etc.)
console.log(JSON.stringify({
...event,
service: 'edge-chat',
env: process.env.VERCEL_ENV,
}));
}Hook it into the middleware after the stream completes. The AI SDK provides usage via the onFinish callback, but on the edge you need to capture it without blocking the response. Use waitUntil (Vercel) or ctx.waitUntil (Cloudflare) to fire-and-forget:
// Inside streamText return, after creating the Response:
export const runtime = 'edge';
export async function POST(req: Request) {
// ... middleware setup ...
const response = await middleware.streamText({ messages });
// Extract usage from the stream without buffering
// This requires the provider to support streaming usage chunks
// (OpenAI does via `stream_options: { include_usage: true }`)
return response;
}For providers that don’t stream usage, approximate with a tokenizer (e.g., tiktoken for OpenAI models) on the prompt and count completion tokens as they stream. It’s not perfect but sufficient for cost dashboards.
Routing directives and client control
Clients should be able to influence routing without server redeploys. Pass directives via headers or request body:
// Client sends: { messages, routing: { prefer: 'speed', maxCostPer1k: 0.001 } }
interface RoutingDirective {
prefer?: 'speed' | 'quality' | 'cost';
maxCostPer1k?: number;
requireCapabilities?: string[]; // e.g., ['vision', 'json-mode']
excludeModels?: string[];
}The middleware evaluates directives when ordering fallbacks:
function orderModels(
primary: ProviderClient,
fallbacks: ProviderClient[],
directive?: RoutingDirective
): ProviderClient[] {
const all = [primary, ...fallbacks];
if (directive?.excludeModels) {
return all.filter(m => !directive.excludeModels!.includes(m.modelId));
}
if (directive?.prefer === 'cost') {
// Sort by known pricing (maintain a pricing map)
return all.sort((a, b) => pricing[a.modelId] - pricing[b.modelId]);
}
if (directive?.prefer === 'speed') {
// Prefer smaller/faster models
return all.sort((a, b) => speedRank[a.modelId] - speedRank[b.modelId]);
}
return all; // Default: primary first, then configured fallback order
}This keeps routing logic centralized. Product teams can experiment with routing strategies via feature flags without backend changes.
Common pitfalls
1. Forgetting export const runtime = 'edge'
Without this, Vercel runs the route on Node.js. The code will work but you lose edge benefits (cold start characteristics, geographic distribution). Always declare the runtime explicitly.
2. Buffering the stream to parse JSON
Some engineers try to JSON.parse the stream to extract tool calls or structured output. This breaks streaming. Use the AI SDK’s streamText with toolChoice and onToolCall callbacks instead — they fire incrementally as the model emits tool calls.
3. Ignoring provider cache headers
Providers return Cache-Control hints (e.g., public, max-age=300 for cached completions). Forward these to the client. If you strip them, you lose the ability to serve cached responses from the edge CDN on repeat requests.
4. Hardcoding model IDs in routes
Routes should reference logical names (chat-default, chat-fast, chat-reasoning). Map these to provider/model strings in a config file. When you want to swap GPT-4o-mini for Claude 3.5 Haiku, you change one line, not every route.
// config/models.ts
export const MODEL_MAP = {
'chat-default': 'openai/gpt-4o-mini',
'chat-fast': 'anthropic/claude-3-haiku',
'chat-reasoning': 'openai/o1-mini',
} as const;5. No request ID propagation
Generate a requestId at the edge entry point (middleware or route) and pass it through every log line, provider call, and response header. Without it, you cannot correlate a user complaint with the provider error log.
const requestId = crypto.randomUUID();
headers.set('x-request-id', requestId);
// Include in all structured logsTesting edge middleware locally
The Vercel CLI (vercel dev) simulates the edge runtime but has limitations: no real geographic distribution, different CPU limits, and no waitUntil emulation. For middleware logic, unit test the fallback ordering and header injection with Vitest:
// lib/edge-middleware.test.ts
import { createMiddleware } from './edge-middleware';
import { createMockProvider } from './test-utils';
describe('edge middleware fallback', () => {
it('falls back on 429', async () => {
const failing = createMockProvider({ error: { status: 429 } });
const succeeding = createMockProvider({ text: 'Hello from fallback' });
const middleware = createMiddleware({
model: 'openai/gpt-4o',
fallbackModels: ['anthropic/claude-3-haiku'],
maxTokens: 100,
temperature: 0,
});
// Inject mock providers (requires refactoring parseModelId to accept overrides)
const response = await middleware.streamText({
messages: [{ role: 'user', content: 'Hi' }]
});
expect(response.headers.get('x-model-used')).toBe('anthropic/claude-3-haiku');
expect(response.headers.get('x-fallback-count')).toBe('1');
});
});Integration test against real providers in CI using a staging environment. Run a smoke test that verifies the full chain: request → edge → provider → stream → client.
When to skip the edge
Edge middleware adds complexity. Stay on Node.js if:
- You need persistent connections (WebSockets, long-polling)
- Your prompt includes large file uploads (> 1 MB) — edge body parsing is limited
- You rely on Node-only packages (native bindings, heavy dependencies)
- Your team lacks observability infrastructure to debug distributed edge failures
The edge shines for: chat completions, structured extraction, classification, and any workload where latency distribution matters more than raw throughput.
Wrapping up
The pattern is straightforward: thin route → middleware handles provider complexity → stream returns to client. The middleware owns fallbacks, usage metering, header forwarding, and routing directives. Your routes stay clean, your clients stay resilient, and your costs stay visible.
Start with the minimal middleware above. Add directives, metering, and timeout handling as real requirements emerge. The edge runtime rewards simplicity — every additional dependency increases cold start variance and failure surface area.