Serverless function timeouts and Vercel AI SDK streaming

Serverless functions and streaming LLM responses are fundamentally at odds. Your cloud provider enforces a hard execution ceiling — 10 seconds on Vercel Hobby, 60 seconds on Pro, 900 seconds on AWS Lambda — while a large model can easily take 30+ seconds to emit a complete answer. The vercel ai sdk serverless timeout streaming problem shows up as truncated responses, 504 errors, or silent failures when the platform kills the function mid-stream. This guide walks through the configuration changes, code patterns, and architectural escapes that keep streams alive across Vercel, AWS Lambda, and Cloudflare Workers.

Step 1: Know your platform limits before you code

Every runtime publishes a maximum invocation duration. Treat these as hard constraints, not suggestions.

Action: Open your provider dashboard and confirm the current limit for the environment you deploy to. Write it down. Every decision below flows from that number.

Step 2: Configure the function timeout in your framework

Vercel (Next.js App Router)

In next.config.js or per-route export const maxDuration = 60:

// next.config.js
module.exports = {
  experimental: {
    serverActions: {
      bodySizeLimit: '2mb',
    },
  },
};

// app/api/chat/route.ts
export const maxDuration = 60; // Pro/Enterprise only

If you are on Hobby, maxDuration is ignored. You must either upgrade or move the long-running work off the request path (see Step 6).

AWS Lambda (SST, CDK, or console)

// stacks/ChatStack.ts
import { Function } from 'sst/constructs';

new Function(this, 'ChatFn', {
  handler: 'src/chat.handler',
  timeout: '5 minutes', // 300 seconds
  runtime: 'nodejs20.x',
  architecture: 'arm_64',
});

Cloudflare Workers (Wrangler)

# wrangler.toml
[build]
command = "npm run build"

[functions]
node_compat = true

# No timeout field — CPU limit is enforced by plan.
# Use Workers AI binding for model inference to avoid CPU accounting.

Step 3: Stream with the Vercel AI SDK correctly

The SDK’s streamText returns a ReadableStream that the runtime must keep alive. Two common mistakes: buffering the entire response before sending, and forgetting to set the Content-Type header that tells the platform this is a streaming response.

// app/api/chat/route.ts (Next.js App Router)
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = await streamText({
    model: openai('gpt-4o'),
    messages,
    // Critical: do NOT await result.text() — that buffers everything.
    // Return the stream directly.
  });

  // This sets Transfer-Encoding: chunked and keeps the function alive
  return result.toDataStreamResponse({
    // Optional: send usage, finish reason, etc. as trailing data
    sendUsage: true,
    sendFinish: true,
  });
}

Verification: curl -N -X POST -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"Write a 500 word essay"}]}' http://localhost:3000/api/chat should emit tokens immediately, not after the full generation completes.

Step 4: Keep the connection alive with heartbeat chunks

Some load balancers (ALB, CloudFront, Vercel Edge) close idle TCP connections after 30–60 seconds of no data. A model thinking silently for 45 seconds looks idle. Send a comment line or a custom keep-alive event every 15 seconds.

import { streamText, StreamData } from 'ai';
import { openai } from '@ai-sdk/openai';

export async function POST(req: Request) {
  const { messages } = await req.json();
  const data = new StreamData();

  const result = streamText({
    model: openai('gpt-4o'),
    messages,
    onFinish: () => data.close(),
  });

  // Heartbeat interval
  const heartbeat = setInterval(() => {
    // SSE comment line — ignored by EventSourceParser, keeps TCP alive
    data.write(':keep-alive\n\n');
  }, 15_000);

  result.consumeStream(); // Start generation

  return result.toDataStreamResponse({
    data,
    sendUsage: true,
  });
}

Verification: Open the network tab, filter for the chat request, and confirm you see :keep-alive\n\n lines every ~15 s until the final chunk arrives.

Step 5: Reduce time-to-first-token and total latency

The fastest way to beat a timeout is to finish sooner. Three levers you control:

5.1 Use a smaller model for the first pass

const result = streamText({
  model: openai('gpt-4o-mini'), // ~2x faster, 1/10th cost
  messages,
  system: 'Answer concisely. If the user needs depth, they will ask.',
});

5.2 Limit maxTokens aggressively

const result = streamText({
  model: openai('gpt-4o'),
  messages,
  maxTokens: 1024, // Hard cap on generation length
});

5.3 Enable provider-side streaming (already default in AI SDK v3+)

The SDK uses stream: true on the provider request. Confirm you are not accidentally setting stream: false in a custom provider wrapper.

Verification: Measure timeToFirstToken in your logs. Target < 800 ms for gpt-4o-mini, < 1.5 s for gpt-4o. If you exceed 5 s, the model or provider is the bottleneck — consider a different endpoint.

Step 6: Move long generations off the request path (the escape hatch)

When the platform limit is lower than your worst-case generation time (Vercel Hobby 10 s, Edge 30 s CPU), you cannot win by tuning. You must decouple generation from the HTTP request.

Pattern: Polling with a job queue

1. Client POSTs /api/chat/start → returns jobId immediately.
2. Background worker (Queue, Upstash, Inngest, or Lambda) runs streamText and writes chunks to Redis / KV / S3.
3. Client polls /api/chat/stream/:jobId via EventSource or fetch, reading stored chunks.
4. When worker finishes, it writes a done: true marker; client closes connection.

// app/api/chat/start/route.ts
import { createClient } from '@vercel/kv';
import { v4 as uuid } from 'uuid';

const kv = createClient();

export async function POST(req: Request) {
  const { messages } = await req.json();
  const jobId = uuid();

  // Fire-and-forget: push to queue (example uses Upstash QStash)
  await fetch('https://qstash.upstash.io/v2/publish/your-worker-url', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.QSTASH_TOKEN}` },
    body: JSON.stringify({ jobId, messages }),
  });

  // Store initial empty state
  await kv.hset(`chat:${jobId}`, { status: 'pending', chunks: '[]' });

  return Response.json({ jobId });
}

// Worker (separate deployment, no timeout pressure)
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { createClient } from '@vercel/kv';

export default async function handler(job: { jobId: string; messages: any[] }) {
  const { jobId, messages } = job;
  const kv = createClient();
  const result = streamText({ model: openai('gpt-4o'), messages });

  for await (const chunk of result.textStream) {
    // Append to Redis list atomically
    await kv.rpush(`chat:${jobId}:chunks`, chunk);
  }

  await kv.hset(`chat:${jobId}`, { status: 'complete' });
}

// app/api/chat/stream/[jobId]/route.ts
import { createClient } from '@vercel/kv';

const kv = createClient();

export async function GET(
  req: Request,
  { params }: { params: { jobId: string } }
) {
  const { jobId } = params;
  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      let lastIndex = 0;
      const interval = setInterval(async () => {
        const chunks = await kv.lrange(`chat:${jobId}:chunks`, lastIndex, -1);
        for (const chunk of chunks) {
          controller.enqueue(encoder.encode(`data: ${chunk}\n\n`));
        }
        lastIndex += chunks.length;

        const status = await kv.hget(`chat:${jobId}`, 'status');
        if (status === 'complete') {
          clearInterval(interval);
          controller.enqueue(encoder.encode('data: [DONE]\n\n'));
          controller.close();
        }
      }, 500);
    },
  });

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      Connection: 'keep-alive',
    },
  });
}

Verification: Start a generation that would take 60 s. Confirm the initial POST returns in < 500 ms, the polling stream emits tokens, and no 504 occurs.

Step 7: Handle client-side reconnection gracefully

Networks flake. Browsers throttle background tabs. Your stream consumer must resume from the last received token.

// components/ChatStream.tsx
'use client';

import { useChat } from 'ai/react';

export function ChatStream() {
  const { messages, input, handleInputChange, handleSubmit, reload, stop } = useChat({
    api: '/api/chat',
    // Critical: send the last message ID so the server can resume
    // Requires server support — see Step 6 pattern or provider `previousMessageId`
    onResponse: (res) => {
      if (!res.ok) {
        // Exponential backoff retry
        setTimeout(() => reload(), 1000);
      }
    },
  });

  return (
    <div>
      {messages.map((m) => (
        <div key={m.id}>{m.content}</div>
      ))}
      <form onSubmit={handleSubmit}>
        <input value={input} onChange={handleInputChange} disabled={messages.length && messages[messages.length - 1].role === 'assistant'} />
        <button type="submit">Send</button>
        <button type="button" onClick={stop}>Stop</button>
      </form>
    </div>
  );
}

Verification: Open the chat, start a long response, disconnect Wi-Fi for 5 s, reconnect. The stream should resume without duplicating or losing tokens.

Step 8: Monitor and alert on timeout-adjacent metrics

You cannot fix what you do not measure. Emit these from every chat invocation:

// lib/telemetry.ts
export function recordChatMetrics({
  model,
  promptTokens,
  completionTokens,
  timeToFirstTokenMs,
  totalDurationMs,
  timedOut,
  platform,
}: {
  model: string;
  promptTokens: number;
  completionTokens: number;
  timeToFirstTokenMs: number;
  totalDurationMs: number;
  timedOut: boolean;
  platform: 'vercel' | 'lambda' | 'cloudflare';
}) {
  // Send to Datadog, Honeycomb, Vercel Analytics, etc.
  fetch('https://api.your-observability.com/v1/events', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      event: 'chat_completion',
      timestamp: Date.now(),
      attributes: { model, promptTokens, completionTokens, timeToFirstTokenMs, totalDurationMs, timedOut, platform },
    }),
    keepalive: true, // Fire-and-forget even if function is shutting down
  }).catch(() => {}); // Swallow — never block the response
}

Dashboards to build:

• P95 totalDurationMs by model and platform
• timedOut rate (target < 0.1 %)
• timeToFirstTokenMs trend (alert if > 2 s sustained)

Step 9: Test the failure modes locally

Do not discover timeouts in production. Simulate them in CI.

# 1. Vercel Hobby limit (10 s)
npx vercel dev --timeout=10

# 2. Lambda limit (configured value)
sam local invoke ChatFn --event events/chat.json --timeout 30

# 3. Cloudflare CPU limit (use `wrangler dev` with --minify to simulate)
wrangler dev --minify

Write an integration test that asserts:

// tests/chat.timeout.test.ts
import { createMocks } from 'node-mocks-http';
import handler from '@/app/api/chat/route';

test('stream completes before platform timeout', async () => {
  const { req, res } = createMocks({
    method: 'POST',
    body: { messages: [{ role: 'user', content: 'x'.repeat(5000) }] },
  });

  await handler(req, res);

  // If the function times out, the test runner will fail with timeout error
  // Set jest timeout slightly above your platform limit
  expect(res._getStatusCode()).toBe(200);
  const chunks = res._getData().split('\n\n').filter(Boolean);
  expect(chunks.length).toBeGreaterThan(1);
}, 55_000); // 55 s for Vercel Pro 60 s limit

Step 10: Document the operational runbook

When (not if) a timeout slips through, the on-call engineer needs a runbook, not a Slack thread.

# Chat Timeout Runbook

## Symptoms
- 504 from Vercel / ALB / CloudFront
- Client shows "Network error" after N seconds
- `timedOut: true` spike in Datadog

## Immediate mitigation
1. Check current platform limit in dashboard (Vercel: Project → Settings → Functions).
2. If on Vercel Hobby → upgrade to Pro or enable queue pattern (Step 6).
3. If on Pro/Enterprise → increase `maxDuration` to 300 s (max).
4. If on Lambda → increase `Timeout` in AWS console / IaC.
5. If on Cloudflare Free → upgrade to Workers Paid or move inference to Workers AI binding.

## Root.

## Root cause investigation
1. Query `chat_completion` events for `model`, `completionTokens`, `totalDurationMs`.
2. Identify if a specific model/provider combo regressed.
3. Check provider status page (OpenAI, Anthropic, etc.) for elevated latency.
4. Verify `maxTokens` guardrail is set (Step 5.2).

## Prevention
- Add CI gate: `p95(totalDurationMs) < 0.8 * platformLimit`
- Alert on `timedOut` rate > 0.1 % for 5 min
- Quarterly load test with 95th-percentile prompt length

The vercel ai sdk serverless timeout streaming conflict is a platform constraint, not a library bug. Configure the function ceiling, stream correctly, heartbeat idle connections, shrink the generation, and when the ceiling is still too low, move the work off the request path. The patterns above have kept production chat endpoints alive across millions of requests on Vercel, Lambda, and Cloudflare. Pick the step that matches your current limit and implement it today.