Vercel AI SDK middleware for logging and caching LLM calls

If you need vercel ai sdk middleware logging caching, the experimental middleware API in the Vercel AI SDK gives you a clean seam to observe and short-circuit model calls. This guide builds a logging middleware and a Redis-backed caching layer that you can drop into any Node or Edge runtime without forking the SDK.

Step 1: Scaffold and install dependencies

Create a minimal TypeScript project. We’ll use pnpm, but npm or yarn work identically.

mkdir ai-mw && cd ai-mw
pnpm init
pnpm add ai @ai-sdk/openai redis zod
pnpm add -D typescript @types/node tsx

The ai package ships the core functions (generateText, streamText) and the experimental_createMiddleware helper. @ai-sdk/openai provides the OpenAI-compatible provider interface. redis is our cache store; swap in ioredis or an in-memory map if you’re prototyping.

Create a tsconfig.json with "module": "ESNext" and "target": "ES2022" so top-level await works.

Step 2: Point the provider at your gateway

Configure the OpenAI provider with a custom base URL. If you route through n4n.ai, its OpenAI-compatible endpoint fronts 240+ models and forwards provider cache-control hints, so the caching middleware below composites with native provider caches.

// lib/provider.ts
import { createOpenAI } from '@ai-sdk/openai';

export const openai = createOpenAI({
  baseURL: process.env.LLM_BASE_URL ?? 'https://api.openai.com/v1',
  apiKey: process.env.LLM_API_KEY!,
});

// Example: model id 'gpt-4o-mini' or any gateway-routed id
export const model = openai('gpt-4o-mini');

Set LLM_BASE_URL and LLM_API_KEY in .env.local. For local dev, leave them blank to hit OpenAI directly.

Step 3: Write a logging middleware

The middleware API exposes lifecycle hooks. onGenerateText fires after a non-streaming completion; onStreamText fires per delta. We capture latency and token usage.

// lib/middleware/logging.ts
import { experimental_createMiddleware } from 'ai';

export const loggingMiddleware = experimental_createMiddleware({
  onGenerateText: async ({ response, startTime }) => {
    const ms = startTime ? Date.now() - startTime : 0;
    console.log('[llm] generate', {
      text: response.text.slice(0, 80),
      ms,
      usage: response.usage,
    });
  },
  onStreamText: async ({ textDelta, startTime }) => {
    if (textDelta) process.stdout.write(textDelta);
  },
  onError: async ({ error }) => {
    console.error('[llm] error', error);
  },
});

startTime is provided by the SDK when the middleware is attached. If you need precise timing, wrap the call yourself—the hook is best-effort.

Step 4: Add caching middleware (write path)

The current experimental_createMiddleware does not let you preempt the model call from inside onGenerateText. The pragmatic pattern: use the hook to write the cache after a miss, and wrap generateText with a read guard. This keeps the cache logic colocated with other middleware.

// lib/middleware/cache.ts
import { experimental_createMiddleware } from 'ai';
import { Redis } from 'redis';
import { createHash } from 'crypto';

const redis = new Redis(process.env.REDIS_URL!);

export function cacheKey(messages: unknown): string {
  return 'llm:' + createHash('sha256').update(JSON.stringify(messages)).digest('hex');
}

export const cachingMiddleware = experimental_createMiddleware({
  onGenerateText: async ({ response, options }) => {
    const key = cacheKey(options.messages);
    await redis.set(key, JSON.stringify(response), 'EX', 3600);
  },
});

// Read guard used before calling generateText
export async function cachedGenerateText(opts: {
  messages: any[];
  model: any;
  middleware?: any[];
}) {
  const key = cacheKey(opts.messages);
  const hit = await redis.get(key);
  if (hit) {
    console.log('[llm] cache hit', key);
    return JSON.parse(hit);
  }
  const { generateText } = await import('ai');
  return generateText({
    model: opts.model,
    messages: opts.messages,
    middleware: [...(opts.middleware ?? []), cachingMiddleware],
  });
}

The options.messages shape is stable across calls for a given prompt, making it a safe cache key. For streaming, you’d cache the assembled text in onStreamText’s final flush instead.

Step 5: Compose middleware in a route

Wire both middlewares into a Next.js App Router handler. Logging runs first, caching writes after the model returns.

// app/api/chat/route.ts
import { loggingMiddleware } from '@/lib/middleware/logging';
import { cachedGenerateText } from '@/lib/middleware/cache';
import { model } from '@/lib/provider';

export async function POST(req: Request) {
  const { messages } = await req.json();
  const result = await cachedGenerateText({
    messages,
    model,
    middleware: [loggingMiddleware],
  });
  return Response.json({ text: result.text });
}

If you later add a second model or a fallback chain, the middleware stays attached because it’s bound at the call site.

Step 6: Verify success

Start Redis and the dev server.

redis-server --daemonize yes
pnpm tsx watch app/api/chat/route.ts # or next dev

Hit the endpoint twice with the same body:

curl -X POST localhost:3000/api/chat -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is a mutex?"}]}'

First call prints [llm] generate with latency and usage, then stores the key. The second call prints [llm] cache hit and returns immediately with no provider request. Confirm with:

redis-cli keys 'llm:*'

You should see one key. Latency on the second call drops to sub-millisecond local Redis read.

Step 7: Production notes

Streaming: Cache the full text in onStreamText by accumulating deltas and writing on the final { isFinal: true } event. Don’t cache partials.

Cache invalidation: Use a namespace prefix per model version (llm:v1:...) so a prompt tweak or model swap busts old entries. Set TTLs aggressively for volatile data.

Token metering: If your gateway emits per-token usage, log it in onGenerateText alongside your own cost tags. The middleware hook receives response.usage directly from the provider.

Error isolation: Wrap Redis calls in try/catch inside the middleware. A cache outage should never block a generation.

Edge runtimes: redis works on Node; on Edge use upstash-redis or Durable Objects. The middleware code stays identical because it only depends on a get/set interface.

The vercel ai sdk middleware logging caching pattern above separates concerns: logging observes, caching short-circuits, and your route stays declarative. Once this is in place, adding rate-limit or PII-redaction middleware is the same shape—implement the hook, push it into the array.