Rate limiting a Vercel AI SDK chatbot in Next.js

Rate limiting a vercel ai sdk chatbot rate limiting next.js implementation protects your API budget and prevents abuse without degrading the experience for legitimate users. The Vercel AI SDK’s streamText and useChat hooks make streaming responses trivial, but they also make it easy for a single client to open dozens of concurrent streams and burn through provider quotas. This guide walks through a production-grade rate limiter built on Upstash Redis, integrated via Next.js middleware, with client-side backoff and observable metrics.

Step 1: Define the rate limit policy

Before writing code, decide what you’re protecting. A typical chatbot has two distinct resources: request count (how many conversations a user starts) and token consumption (how much model output they generate). For most teams, a sliding window on request count is the right starting point — it’s predictable, easy to explain, and maps cleanly to provider rate limits.

// lib/rate-limit-policy.ts
export const RATE_LIMIT_POLICY = {
  // Anonymous users: 10 requests per minute
  anonymous: { requests: 10, windowMs: 60_000 },
  // Authenticated users: 60 requests per minute
  authenticated: { requests: 60, windowMs: 60_000 },
  // Premium tier: 300 requests per minute
  premium: { requests: 300, windowMs: 60_000 },
} as const;

export type Tier = keyof typeof RATE_LIMIT_POLICY;

This policy lives in a shared module so both middleware and API routes reference the same numbers. Adjust the tiers to match your pricing or auth model.

Step 2: Set up Upstash Redis

Upstash provides a serverless Redis with HTTP/REST access — no connection pooling, no VPC config, and a generous free tier. Install the client:

npm install @upstash/redis

Create a singleton client. The Upstash SDK reads UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN from environment variables automatically.

// lib/redis.ts
import { Redis } from '@upstash/redis';

export const redis = new Redis({
  url: process.env.UPSTASH_REDIS_REST_URL!,
  token: process.env.UPSTASH_REDIS_REST_TOKEN!,
});

Add the credentials to .env.local (never commit these):

UPSTASH_REDIS_REST_URL=https://your-instance.upstash.io
UPSTASH_REDIS_REST_TOKEN=your-token

Verify connectivity with a quick script:

node -e "require('dotenv').config(); const {redis} = require('./lib/redis'); redis.ping().then(console.log).catch(console.error)"

You should see PONG.

Step 3: Build the sliding window algorithm

A sliding window uses a sorted set where the score is the request timestamp. This gives exact limits without the boundary artifacts of fixed windows.

// lib/rate-limiter.ts
import { redis } from './redis';
import { RATE_LIMIT_POLICY, type Tier } from './rate-limit-policy';

export interface RateLimitResult {
  allowed: boolean;
  remaining: number;
  resetMs: number;
  totalRequests: number;
}

const WINDOW_MS = 60_000; // 1 minute base window

export async function checkRateLimit(
  identifier: string,
  tier: Tier = 'anonymous'
): Promise<RateLimitResult> {
  const { requests: maxRequests, windowMs } = RATE_LIMIT_POLICY[tier];
  const now = Date.now();
  const windowStart = now - windowMs;
  const key = `ratelimit:${tier}:${identifier}`;

  // Remove expired entries, add current request, count, set TTL
  const pipeline = redis.pipeline();
  pipeline.zremrangebyscore(key, 0, windowStart);
  pipeline.zadd(key, { score: now, member: `${now}-${Math.random()}` });
  pipeline.zcard(key);
  pipeline.pexpire(key, windowMs);
  const results = await pipeline.exec();

  const totalRequests = results[2] as number;
  const allowed = totalRequests <= maxRequests;
  const remaining = Math.max(0, maxRequests - totalRequests);
  const resetMs = now + windowMs;

  return { allowed, remaining, resetMs, totalRequests };
}

The Math.random() suffix on the member ensures uniqueness when multiple requests arrive in the same millisecond. The pipeline executes atomically, so you never race between cleanup and counting.

Step 4: Identify the client in middleware

Next.js middleware runs before your API routes, making it the natural place to enforce limits. You need a stable identifier per client. For authenticated users, use their user ID. For anonymous users, fall back to a hashed IP + user agent fingerprint.

// middleware.ts
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';
import { checkRateLimit } from './lib/rate-limiter';
import { getTierFromSession } from './lib/auth-helpers'; // your auth logic

export async function middleware(request: NextRequest) {
  // Only protect chat API routes
  if (!request.nextUrl.pathname.startsWith('/api/chat')) {
    return NextResponse.next();
  }

  const tier = await getTierFromSession(request);
  const identifier = getClientIdentifier(request, tier);

  const result = await checkRateLimit(identifier, tier);

  const response = result.allowed
    ? NextResponse.next()
    : new NextResponse(JSON.stringify({ error: 'Rate limit exceeded' }), {
        status: 429,
        headers: { 'Content-Type': 'application/json' },
      });

  // Standard rate limit headers
  response.headers.set('X-RateLimit-Limit', RATE_LIMIT_POLICY[tier].requests.toString());
  response.headers.set('X-RateLimit-Remaining', result.remaining.toString());
  response.headers.set('X-RateLimit-Reset', new Date(result.resetMs).toISOString());

  return response;
}

function getClientIdentifier(request: NextRequest, tier: string): string {
  if (tier !== 'anonymous') {
    return `user:${tier}`; // getTierFromSession returns user ID for authed tiers
  }
  const ip = request.headers.get('x-forwarded-for')?.split(',')[0]?.trim() ||
             request.headers.get('x-real-ip') ||
             'unknown';
  const ua = request.headers.get('user-agent') || 'unknown';
  // Hash to avoid storing raw PII in Redis
  return `anon:${hashString(ip + '|' + ua)}`;
}

function hashString(input: string): string {
  let hash = 0;
  for (let i = 0; i < input.length; i++) {
    hash = ((hash << 5) - hash) + input.charCodeAt(i);
    hash |= 0;
  }
  return Math.abs(hash).toString(36);
}

export const config = {
  matcher: '/api/chat/:path*',
};

The matcher ensures middleware only runs on your chat endpoints. The X-RateLimit-* headers follow the de facto standard (GitHub, Twitter, etc.) so clients can build retry logic without guessing.

Step 5: Handle 429 in the API route

Middleware returns a 429 before your route handler executes, but you still need the route to exist for non-rate-limited requests and for local development where middleware may not run.

// app/api/chat/route.ts
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { NextRequest } from 'next/server';

export async function POST(request: NextRequest) {
  const { messages } = await request.json();

  const result = await streamText({
    model: openai('gpt-4o-mini'),
    messages,
    maxTokens: 500,
    temperature: 0.7,
  });

  return result.toDataStreamResponse();
}

This stays clean — all rate limit logic lives in middleware. If you need per-route customization (e.g., stricter limits on /api/chat/completions vs /api/chat/embeddings), add a route segment check in middleware or call checkRateLimit directly in the handler with a different key prefix.

Step 6: Client-side retry with exponential backoff

The Vercel AI SDK’s useChat hook accepts an onError callback and a custom fetch implementation. Wrap fetch to respect Retry-After or X-RateLimit-Reset headers and back off.

// lib/fetch-with-backoff.ts
type FetchWithBackoffOptions = {
  maxRetries?: number;
  baseDelayMs?: number;
};

export function createFetchWithBackoff(options: FetchWithBackoffOptions = {}) {
  const { maxRetries = 3, baseDelayMs = 1000 } = options;

  return async function fetchWithBackoff(
    input: RequestInfo | URL,
    init?: RequestInit
  ): Promise<Response> {
    let attempt = 0;

    while (true) {
      const response = await fetch(input, init);

      if (response.status !== 429 || attempt >= maxRetries) {
        return response;
      }

      // Prefer Retry-After header (seconds), fall back to X-RateLimit-Reset
      const retryAfter = response.headers.get('Retry-After');
      const resetHeader = response.headers.get('X-RateLimit-Reset');
      let delayMs: number;

      if (retryAfter) {
        delayMs = parseInt(retryAfter, 10) * 1000;
      } else if (resetHeader) {
        delayMs = new Date(resetHeader).getTime() - Date.now();
      } else {
        delayMs = baseDelayMs * Math.pow(2, attempt);
      }

      delayMs = Math.max(0, Math.min(delayMs, 30_000)); // cap at 30s
      await sleep(delayMs);
      attempt++;
    }
  };
}

function sleep(ms: number): Promise<void> {
  return new Promise(resolve => setTimeout(resolve, ms));
}

Wire it into useChat:

// components/Chat.tsx
'use client';

import { useChat } from 'ai/react';
import { createFetchWithBackoff } from '@/lib/fetch-with-backoff';

export function Chat() {
  const { messages, input, handleInputChange, handleSubmit, isLoading, error } = useChat({
    api: '/api/chat',
    fetch: createFetchWithBackoff({ maxRetries: 3, baseDelayMs: 1500 }),
    onError: (err) => {
      if (err instanceof Error && err.message.includes('429')) {
        // Toast or inline notice: "Slow down, you're sending messages too quickly"
        console.warn('Rate limited — backing off');
      }
    },
  });

  return (
    <div>
      {messages.map(m => (
        <div key={m.id} className={m.role}>
          {m.content}
        </div>
      ))}
      {isLoading && <span className="typing-indicator">…</span>}
      <form onSubmit={handleSubmit}>
        <input
          value={input}
          onChange={handleInputChange}
          placeholder="Type a message…"
          disabled={isLoading}
        />
        <button type="submit" disabled={isLoading || !input.trim()}>
          Send
        </button>
      </form>
      {error && <div className="error">{error.message}</div>}
    </div>
  );
}

The custom fetch retries transparently. The user sees a brief pause, then the stream resumes. For hard limits (premium tier exhausted), onError lets you surface a friendly upgrade prompt.

Step 7: Token-based limiting (optional but recommended)

Request counting alone doesn’t stop a user from asking “write me a 10,000 word essay” in one message. If you’re paying per output token, add a token budget that decrements on each completion.

// lib/token-budget.ts
import { redis } from './redis';

const TOKEN_WINDOW_MS = 60_000;
const TOKEN_LIMITS = {
  anonymous: 5_000,
  authenticated: 50_000,
  premium: 500_000,
} as const;

export async function consumeTokenBudget(
  identifier: string,
  tier: keyof typeof TOKEN_LIMITS,
  estimatedTokens: number
): Promise<{ allowed: boolean; remaining: number }> {
  const key = `tokenbudget:${tier}:${identifier}`;
  const limit = TOKEN_LIMITS[tier];
  const now = Date.now();
  const windowStart = now - TOKEN_WINDOW_MS;

  const pipeline = redis.pipeline();
  pipeline.zremrangebyscore(key, 0, windowStart);
  pipeline.zadd(key, { score: now, member: `${now}-${Math.random()}`, value: estimatedTokens });
  pipeline.zrange(key, 0, -1, { withScores: true });
  pipeline.pexpire(key, TOKEN_WINDOW_MS);
  const results = await pipeline.exec();

  // Sum token values in current window
  const entries = results[2] as Array<{ score: number; value: string }>;
  const used = entries.reduce((sum, e) => sum + parseInt(e.value, 10), 0);
  const allowed = used <= limit;
  const remaining = Math.max(0, limit - used);

  return { allowed, remaining };
}

Call this from your API route after streamText resolves, using the actual usage.completionTokens from the result. If the budget is exceeded, return a 429 with a distinct error code so the client can show “Token budget exceeded — upgrade for more.”

// app/api/chat/route.ts (excerpt)
import { consumeTokenBudget } from '@/lib/token-budget';
import { getTierFromSession } from '@/lib/auth-helpers';

export async function POST(request: NextRequest) {
  const { messages } = await request.json();
  const tier = await getTierFromSession(request);
  const identifier = getIdentifierFromRequest(request); // same logic as middleware

  const result = await streamText({
    model: openai('gpt-4o-mini'),
    messages,
    maxTokens: 500,
  });

  // Consume budget after we know actual usage
  const { usage } = await result.consumeStream(); // waits for stream to finish
  const { allowed, remaining } = await consumeTokenBudget(identifier, tier, usage.completionTokens);

  if (!allowed) {
    return new NextResponse(JSON.stringify({ error: 'Token budget exceeded', code: 'TOKEN_BUDGET_EXCEEDED' }), {
      status: 429,
      headers: { 'Content-Type': 'application/json', 'X-RateLimit-Remaining-Tokens': remaining.toString() },
    });
  }

  return result.toDataStreamResponse();
}

Note: consumeStream() buffers the full response. For true streaming with token accounting, you’d need a custom stream transformer that counts tokens on the fly — more complex, but doable with Tiktoken.

Step 8: Observability and alerting

Rate limits are useless if you don’t know when they trigger. Emit structured logs for every 429 and for near-limit warnings (e.g., >80% utilization).

// lib/rate-limit-metrics.ts
import { redis } from './redis';

export async function recordRateLimitEvent(params: {
  identifier: string;
  tier: string;
  allowed: boolean;
  remaining: number;
  limit: number;
  endpoint: string;
}) {
  const { allowed, remaining, limit, ...rest } = params;
  const utilization = 1 - remaining / limit;

  // Structured log for your log aggregator (Datadog, Axiom, etc.)
  console.log(JSON.stringify({
    event: 'rate_limit_check',
    allowed,
    utilization: Math.round(utilization * 100),
    ...rest,
    timestamp: new Date().toISOString(),
  }));

  // Increment a counter for alerting
  if (!allowed) {
    await redis.incr(`metrics:ratelimit:denied:${params.tier}:${params.endpoint}`);
  } else if (utilization > 0.8) {
    await redis.incr(`metrics:ratelimit:near_limit:${params.tier}:${params.endpoint}`);
  }
}

Call this from middleware after checkRateLimit. Build a dashboard on the metrics:ratelimit:* keys to spot abuse patterns or misconfigured tiers.

Step 9: Verify the implementation

Run the full flow locally and in staging:

Unit test the limiter — feed it rapid calls and assert the count matches RATE_LIMIT_POLICY[tier].requests.

// lib/rate-limiter.test.ts
import { checkRateLimit } from './rate-limiter';
import { redis } from './redis';

beforeEach(async () => {
  await redis.flushall();
});

test('allows up to limit then denies', async () => {
  const identifier = 'test-user';
  for (let i = 0; i < 10; i++) {
    const result = await checkRateLimit(identifier, 'anonymous');
    expect(result.allowed).toBe(true);
    expect(result.remaining).toBe(9 - i);
  }
  const denied = await checkRateLimit(identifier, 'anonymous');
  expect(denied.allowed).toBe(false);
  expect(denied.remaining).toBe(0);
});

Integration test the middleware — use next-test-api-route-handler or hit the dev server with curl:

for i in {1..12}; do
  curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3000/api/chat \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"hi"}]}'
done
# Expect: ten 200s, then 429s

Load test — run hey or k6 against a staging deployment to confirm Redis doesn’t become a bottleneck. Upstash handles thousands of ops/sec; the pipeline keeps latency under 5ms p99.
Verify headers — check that X-RateLimit-Remaining decrements and X-RateLimit-Reset moves forward.
Test client backoff — artificially lower the limit to 1, send two rapid messages from the UI, and confirm the second retries and succeeds after the window rolls over.

Step 10: Harden for production

A few operational details separate a prototype from a system you can page on:

Rotate Redis credentials quarterly. Upstash supports multiple tokens; add a new one, deploy, revoke the old.
Set a budget alert on your Upstash usage — a runaway loop can spike Redis costs.
Add a bypass header for internal tooling: if (request.headers.get('x-internal-bypass') === process.env.INTERNAL_BYPASS_SECRET) return NextResponse.next();
Document the tiers in your API docs so frontend teams know the contract.
Consider geographic sharding if you serve global traffic — Upstash has regional endpoints; pick the one nearest your Vercel region (usually iad1 for US East).

Where n4n.ai fits

If your chatbot fans out to multiple model providers (OpenAI, Anthropic, Cohere, open-source via Together, etc.), you’re already managing multiple rate limit surfaces. n4n.ai consolidates that behind one OpenAI-compatible endpoint with automatic fallback when a provider is rate-limited or degraded, and it forwards provider cache-control hints so you can respect upstream limits without duplicating logic. The rate limiter you just built still applies at your gateway layer — n4n.ai handles the downstream complexity.

You now have a rate limiter that survives traffic spikes, gives clients predictable headers, and emits the observability you need to tune tiers over time. The same pattern extends to WebSocket connections, webhook ingestion, or any endpoint where “too many requests” means real money.