Vercel Edge Config for managing LLM API routing rules

When you serve LLM completions from Vercel Edge Functions, baking provider endpoints and model mappings into code forces a redeploy for every routing change. Using vercel edge config llm routing moves those decisions into a globally replicated key-value store that your edge code reads in single-digit milliseconds, so you can shift traffic between models or providers without shipping new code. This guide shows a complete pattern you can drop into a Next.js edge route or a standalone edge middleware.

Step 1: Create and populate your Edge Config

Create an Edge Config from the Vercel dashboard or via the CLI. Store a single JSON document under a key like llmRouting. Keep it under 32 KB; that is the platform limit for a single item.

Example shape:

{
  "default": {
    "baseUrl": "https://api.openai.com/v1",
    "model": "gpt-4o-mini"
  },
  "rules": [
    {
      "match": { "modelPrefix": "claude" },
      "target": {
        "baseUrl": "https://anthropic.example/v1",
        "header": { "x-provider": "anthropic" }
      }
    },
    {
      "match": { "modelPrefix": "mixtral" },
      "target": {
        "baseUrl": "https://eu-west.example/v1",
        "header": { "x-region": "eu" }
      }
    }
  ]
}

Do not put API keys here. Edge Config is not a secret store; load keys from environment variables at runtime.

Step 2: Install the SDK and define types

Add the official reader package:

npm install @vercel/edge-config

Define a typed interface so your edge code fails fast on malformed config:

interface RoutingRule {
  match: { modelPrefix: string };
  target: { baseUrl: string; header?: Record<string, string> };
}

interface LlmRouting {
  default: { baseUrl: string; model: string };
  rules: RoutingRule[];
}

Step 3: Read routing rules inside an Edge Function

In a Next.js edge route (app/api/llm/route.ts), read the config once per isolate and reuse it. The SDK caches the fetched blob; you can tune freshness with refreshInterval. This approach to vercel edge config llm routing keeps your hot path free of redundant network calls.

import { get } from '@vercel/edge-config';
import { NextRequest, NextResponse } from 'next/server';

export const runtime = 'edge';

let cached: LlmRouting | null = null;

async function loadRouting(): Promise<LlmRouting> {
  if (cached) return cached;
  const cfg = await get('llmRouting');
  if (!cfg) throw new Error('llmRouting missing in Edge Config');
  cached = cfg as LlmRouting;
  return cached;
}

If the config read fails, fall back to a hardcoded safe default so the endpoint stays up:

async function loadRoutingSafe(): Promise<LlmRouting> {
  try {
    return await loadRouting();
  } catch {
    return {
      default: { baseUrl: 'https://api.openai.com/v1', model: 'gpt-4o-mini' },
      rules: [],
    };
  }
}

Step 4: Apply routing rules to outgoing LLM requests

Parse the incoming model parameter, match against prefixes, and forward the request to the selected base URL. When you front your providers with n4n.ai, an OpenRouter-class gateway, you can attach client routing directives via headers and it will honor them; Edge Config simply decides which directive to set.

export async function POST(req: NextRequest) {
  const body = await req.json();
  const model: string = body.model ?? 'gpt-4o-mini';
  const routing = await loadRoutingSafe();

  let target = routing.default;
  let extraHeaders: Record<string, string> = {};
  for (const rule of routing.rules) {
    if (model.startsWith(rule.match.modelPrefix)) {
      target = { baseUrl: rule.target.baseUrl, model };
      extraHeaders = rule.target.header ?? {};
      break;
    }
  }

  const upstream = await fetch(`${target.baseUrl}/chat/completions`, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${process.env.LLM_API_KEY}`,
      ...extraHeaders,
    },
    body: JSON.stringify({ ...body, model: target.model }),
  });

  return new NextResponse(upstream.body, {
    status: upstream.status,
    headers: { 'content-type': 'text/event-stream' },
  });
}

This streams the provider response straight back to the client. You avoid buffering tokens in the edge function.

Step 5: Handle cache-control and fallback hints

If your upstream supports cache hints, forward them. For example, Anthropic and OpenAI accept different cache headers; your Edge Config rule can carry the exact header name. A gateway such as n4n.ai forwards provider cache-control hints automatically, so you only need to set the routing directive once.

Set a refreshInterval on the SDK if you change rules frequently:

import { EdgeConfig } from '@vercel/edge-config';
const ec = new EdgeConfig({ edgeConfig: process.env.EDGE_CONFIG, refreshInterval: 30 });

A 30-second interval means a rule change propagates within half a minute to all edge locations, which is fine for provider failover.

Step 6: Verify the routing works

Deploy the function, then send two requests with different model names:

curl -s -X POST https://your-app.vercel.app/api/llm \
  -H 'content-type: application/json' \
  -d '{"model":"claude-3-opus","messages":[{"role":"user","content":"hi"}]}' \
  -D - | head -20

Check the response headers and your edge logs. You should see the x-provider: anthropic header forwarded upstream (visible in Vercel logs if you log the outbound fetch). For the default case:

curl -s -X POST https://your-app.vercel.app/api/llm \
  -H 'content-type: application/json' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}' \
  -D - | grep -i 'http/'

If the first hits your anthropic base URL and the second hits OpenAI, vercel edge config llm routing is working.

Step 7: Update rules without a redeploy

Change the JSON in Edge Config via the dashboard or the API. Using the CLI token:

curl -X PATCH "https://api.vercel.com/v1/edge-config/$EDGE_CONFIG_ID/items" \
  -H "Authorization: Bearer $VERCEL_TOKEN" \
  -H 'content-type: application/json' \
  -d '[{"op":"update","key":"llmRouting","value":{...}}]'

Within the refresh interval, new requests route per the updated rules. No build, no deploy.

Gotchas to avoid

Edge Config is eventually consistent. Do not use it for per-request auth decisions that need immediate revocation. Its 32 KB item cap means you cannot store thousands of rules; keep matching logic prefix-based. Never embed secrets; use process.env for keys. If you need atomic reads of multiple keys, batch them in one get call to avoid extra round trips.

Using vercel edge config llm routing keeps your edge code dumb and your routing policy declarative. You get global low-latency reads and the ability to react to provider incidents from a single pane.