AWS Lambda function URLs for low-latency LLM proxying

AWS Lambda function URLs give you a single HTTPS endpoint per function without standing up API Gateway. An aws lambda function urls llm proxy lets you front an LLM provider with custom auth, caching, and routing while keeping cold starts under control. This guide walks through a production-grade deployment on Lambda using Function URLs, Node.js 20, and the built-in fetch API.

Step 1: Scaffold the streaming proxy handler

Create a project directory and write a handler that runs in Lambda’s RESPONSE_STREAM mode. The handler receives the raw HTTP event, checks a shared token, forwards the body to an OpenAI-compatible upstream, and pipes the upstream stream back to the client.

mkdir llm-proxy && cd llm-proxy
npm init -y

Write index.mjs:

export const handler = async (event, context, responseStream) => {
  const method = event.requestContext.http.method;
  if (method !== 'POST') {
    responseStream.writeHead(405, { 'content-type': 'application/json' });
    responseStream.end(JSON.stringify({ error: 'Method not allowed' }));
    return;
  }

  // Function URLs lowercase all header keys
  const auth = event.headers['x-api-key'];
  if (auth !== process.env.PROXY_TOKEN) {
    responseStream.writeHead(401, { 'content-type': 'application/json' });
    responseStream.end(JSON.stringify({ error: 'Unauthorized' }));
    return;
  }

  const body = JSON.parse(event.body || '{}');
  const upstream = process.env.UPSTREAM_URL || 'https://api.openai.com/v1/chat/completions';

  try {
    const upstreamRes = await fetch(upstream, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        'authorization': `Bearer ${process.env.UPSTREAM_KEY}`,
      },
      body: JSON.stringify(body),
    });

    responseStream.writeHead(upstreamRes.status, {
      'content-type': upstreamRes.headers.get('content-type') || 'application/json',
      'cache-control': upstreamRes.headers.get('cache-control') || 'no-store',
    });

    const reader = upstreamRes.body.getReader();
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      responseStream.write(Buffer.from(value));
    }
    responseStream.end();
  } catch (err) {
    responseStream.writeHead(502, { 'content-type': 'application/json' });
    responseStream.end(JSON.stringify({ error: 'Upstream error', detail: err.message }));
  }
};

The code copies the upstream cache-control header through. That matters if you later place a CDN in front and want provider cache hints to survive the proxy. Lambda Function URLs deliver event.headers with all keys lowercased, so we read x-api-key exactly as shown.

Why RESPONSE_STREAM matters

Function URLs support BUFFERED (default) and RESPONSE_STREAM invoke modes. Buffered mode waits for the full upstream response before returning anything, which destroys interactivity for token-by-token LLM output. RESPONSE_STREAM writes chunks to the client as they arrive from the model.

Step 2: Package and deploy the function

Zip the source and create the Lambda function. Use the Node.js 20 runtime; it ships native fetch and streams.

zip -r function.zip index.mjs package.json

aws lambda create-function \
  --function-name llm-proxy \
  --runtime nodejs20.x \
  --handler index.handler \
  --zip-file fileb://function.zip \
  --role arn:aws:iam::123456789012:role/lambda-exec \
  --memory-size 256 \
  --timeout 30 \
  --environment "Variables={PROXY_TOKEN=secret123,UPSTREAM_URL=https://api.openai.com/v1/chat/completions,UPSTREAM_KEY=sk-...}"

The execution role needs only logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents. Memory at 256 MB is sufficient for a pass-through proxy; Lambda allocates CPU proportionally, so bump to 512 MB if you add JSON transformation or logging overhead.

Step 3: Enable the Function URL with streaming

Create the URL config and explicitly set --invoke-mode RESPONSE_STREAM. Choose NONE if you rely on the in-code token check, or AWS_IAM to require SigV4 signing.

aws lambda create-function-url-config \
  --function-name llm-proxy \
  --auth-type NONE \
  --invoke-mode RESPONSE_STREAM

If you expose it with NONE, lock it down with a resource policy that restricts source IP ranges or a WAF ACL. Retrieve the endpoint:

aws lambda get-function-url-config --function-name llm-proxy --query FunctionUrl

The returned URL looks like https://<id>.lambda-url.<region>.on.aws/. It is regional and uses AWS-managed TLS.

Step 4: Point at a multi-provider gateway (optional)

Writing retry and fallback logic inside Lambda is doable but adds latency and complexity. If you want automatic fallback when a provider is rate-limited or degraded, point UPSTREAM_URL at a gateway that already does this. n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models with automatic fallback and per-token metering, so the proxy above works unchanged by swapping the environment variable.

aws lambda update-function-configuration \
  --function-name llm-proxy \
  --environment "Variables={PROXY_TOKEN=secret123,UPSTREAM_URL=https://api.n4n.ai/v1/chat/completions,UPSTREAM_KEY=your-gateway-key}"

Because the handler forwards the request body verbatim and copies headers, client-side routing directives or provider cache-control hints pass through without extra code.

Step 5: Tune for low latency

Cold starts are the main latency tax. Keep them small:

• The deployment package is a few kilobytes; no npm dependencies.
• Native fetch reuses TCP connections across warm invocations within the same execution environment.
• Set timeout to 30 seconds; most chat completions finish well under that.
• Deploy in the region closest to your users; Function URLs have no global accelerator built in.

If you need consistently low time-to-first-byte, warm the function with a CloudWatch Events rule firing every minute:

aws events put-rule --schedule-expression "rate(1 minute)" --name warm-llm-proxy
aws events put-targets --rule warm-llm-proxy --targets "Id=1,Arn=arn:aws:lambda:us-east-1:123456789012:function:llm-proxy"

A single concurrent invocation keeps one environment warm; subsequent calls reuse the V8 isolate and the kept-alive connection to upstream.

Step 6: Add CORS for browser clients

If a browser app calls the proxy directly, the Function URL must return CORS headers. Extend the handler’s writeHead calls to include them:

const cors = {
  'access-control-allow-origin': '*',
  'access-control-allow-headers': 'content-type,x-api-key',
  'access-control-allow-methods': 'POST,OPTIONS',
};
// merge into writeHead objects

Handle OPTIONS preflight in the method check:

if (method === 'OPTIONS') {
  responseStream.writeHead(204, cors);
  responseStream.end();
  return;
}

Step 7: Verify end to end

Call the URL with curl -N to disable client buffering and watch tokens arrive live.

curl -N -X POST https://<id>.lambda-url.<region>.on.aws/ \
  -H "x-api-key: secret123" \
  -H "content-type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi in 5 words"}]}'

Success criteria:

• HTTP 200, upstream content-type preserved.
• Tokens stream incrementally, not after a multi-second pause.
• Omitting x-api-key returns 401.
• GET returns 405.

Measure time to first byte:

curl -N -o /dev/null -w "TTFB: %{time_starttransfer}s\n" -X POST <url> -H "x-api-key: secret123" -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

If you see a delay equal to full generation before any output, the URL is in BUFFERED mode. Delete and recreate the config with --invoke-mode RESPONSE_STREAM.

Step 8: Route and cache directives

A common reason to run an aws lambda function urls llm proxy is to sit in front of multiple models. Add a routing header that rewrites the target model:

const routing = event.headers['x-model-routing'];
const outBody = { ...body };
if (routing) outBody.model = routing;

Forward any client routing headers untouched so downstream gateways can honor them. The proxy stays transparent to cache logic because we already copy cache-control from the upstream response.

Operational guardrails

Function URLs inherit Lambda concurrency limits but have no built-in usage plans. Set reserved concurrency to cap spend:

aws lambda put-function-concurrency --function-name llm-proxy --reserved-concurrent-executions 10

Put CloudFront in front if you need edge TLS termination or to cache OPTIONS responses. For public exposure, attach AWS WAF with a rate-based rule. The aws lambda function urls llm proxy pattern costs only execution time; at 256 MB and ~200 ms average, a million calls is a few dollars.

Verification checklist

• Lambda created with Node 20 and RESPONSE_STREAM URL config.
• curl without token gets 401; wrong method gets 405.
• Streaming confirmed with curl -N and TTFB under a second for small prompts.
• Upstream swapped to multi-provider gateway with no code change (if used).
• CloudWatch shows no Buffer limit errors.

That is the full path from empty repo to a working low-latency LLM proxy on Lambda Function URLs.