Streaming LLM responses from AWS Lambda with response streaming

AWS Lambda response streaming llm output changes the game for serverless chat backends: you no longer buffer the entire completion before returning it to the client. By using Lambda’s response streaming payload format and an OpenAI-compatible client, you can pipe tokens to the browser within milliseconds of generation. Previously you needed API Gateway WebSockets or client polling to fake this; now a plain HTTP function URL does it.

Step 1: Provision a Lambda runtime that supports response streaming

Lambda added response streaming for the Python 3.12 runtime via a writable stream object exposed on the invocation context. You do not need a custom runtime, but you must invoke the function through a Lambda Function URL or an HTTP API explicitly configured for streaming. Standard synchronous invocation always buffers.

Create the function with the managed Python 3.12 runtime:

aws lambda create-function \
  --function-name llm-streamer \
  --runtime python3.12 \
  --handler handler.handler \
  --role arn:aws:iam::123456789012:role/lambda-exec \
  --zip-file fileb://deploy.zip \
  --timeout 60

The execution role needs only basic Lambda permissions. No VPC is required unless your model provider sits in a private network. Keep the timeout at 60 seconds or higher; streaming does not shorten generation time, it only improves perceived latency.

Step 2: Pin dependencies and package the handler

The OpenAI Python SDK is the path of least resistance because it speaks the OpenAI-compatible HTTP protocol and parses Server-Sent Events (SSE) client-side. Create a requirements.txt:

openai==1.40.0

Install into a local directory and zip with your handler:

pip install -r requirements.txt --target package --no-cache-dir
cp handler.py package/
cd package && zip -r ../deploy.zip . && cd ..

The zip layout should place handler.py at the root next to openai/ and httpx/. A 5–8 MB package is normal and keeps cold starts under a second on Python 3.12.

Step 3: Implement the streaming handler

The handler must grab the response stream from context, write a valid HTTP status line and headers, then iterate the LLM stream and write frames. Using text/event-stream keeps the output compatible with browser EventSource or fetch readers.

import os
import json
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("OPENAI_BASE_URL"),  # optional gateway
)

def handler(event, context):
    stream = context.get_response_stream()
    stream.write(b"HTTP/1.1 200 OK\n")
    stream.write(b"Content-Type: text/event-stream\n")
    stream.write(b"Cache-Control: no-cache\n\n")

    body = json.loads(event.get("body", "{}"))
    prompt = body.get("prompt", "Say hello.")

    try:
        completion = client.chat.completions.create(
            model=os.environ.get("MODEL", "gpt-3.5-turbo"),
            messages=[{"role": "user", "content": prompt}],
            stream=True,
            stream_options={"include_usage": True},
        )
        for chunk in completion:
            data = chunk.model_dump_json()
            stream.write(f"data: {data}\n\n".encode("utf-8"))
        stream.write(b"data: [DONE]\n\n")
    except Exception as e:
        err = json.dumps({"error": str(e)})
        stream.write(f"data: {err}\n\n".encode("utf-8"))
    finally:
        stream.close()

If you want only the delta text instead of the full chunk, replace the inner write with chunk.choices[0].delta.content or "". If you’d rather not manage multiple provider keys, point base_url at an OpenAI-compatible gateway such as n4n.ai, which addresses 240+ models and automatically falls back when a provider is rate-limited. The client code stays identical.

Step 4: Enable streaming on the Lambda Function URL

A standard Lambda invocation buffers the response. You must create a function URL with RESPONSE_STREAM invoke mode:

aws lambda create-function-url-config \
  --function-name llm-streamer \
  --auth-type NONE \
  --invoke-mode RESPONSE_STREAM \
  --cors "AllowOrigins=*"

Note the returned URL. For production, switch --auth-type to AWS_IAM or place CloudFront in front with an authorizer. CORS is required if a browser calls the URL directly.

Step 5: Deploy the packaged code

Upload the zip and set environment variables. Use Secrets Manager for the key in real deployments; plaintext env vars are fine for a local test.

aws lambda update-function-code \
  --function-name llm-streamer \
  --zip-file fileb://deploy.zip

aws lambda update-function-configuration \
  --function-name llm-streamer \
  --environment "Variables={OPENAI_API_KEY=sk-...,MODEL=gpt-3.5-turbo}"

Lambda streams bytes as you write them, but the cumulative response still counts against the 20 MB limit. For long completions, cap output server-side or summarize.

Step 6: Verify streaming with curl and Python

Point curl at the function URL with -N to disable buffering:

curl -N -X POST https://<url>.lambda-url.us-east-1.on.aws/ \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Explain TCP fast open in one sentence."}'

You should see SSE frames arrive incrementally:

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"TCP"}}]}
data: {"id":"chatcmpl-...","choices":[{"delta":{"content":" fast"}}]}
data: [DONE]

If you get the full body after a delay instead of line-by-line output, the invoke mode is wrong or you are behind a proxy that buffers. Use the Lambda URL directly for this test. A Python client confirms the same:

import requests
r = requests.post(URL, json={"prompt":"Hi"}, stream=True)
for line in r.iter_lines():
    if line:
        print(line.decode())

Step 7: Handle partial failures mid-stream

Providers can fail after the header is sent. Because the HTTP status is already 200, signal errors inside the event stream. The try/except in Step 3 writes an error frame. On the client, treat any data: frame containing an error key as terminal and close the connection.

Do not attempt to change the HTTP status code after the first byte is written; Lambda raises a stream-consume error. Design clients to replay the conversation if a stream dies early rather than expecting resume.

Step 8: Production considerations

Cold starts add 200–800 ms to the first token depending on package size. Trim the zip by removing unused SDK extras. Each open stream holds a Lambda invocation and a connection; monitor ConcurrentExecutions and set reserved concurrency to protect downstream provider rate limits.

If you use an OpenAI-compatible gateway, per-token usage metering arrives in the final usage chunk when stream_options={"include_usage": True} is set. Forward that to your billing pipeline. Gateways like n4n.ai forward provider cache-control hints, so set the same Anthropic or OpenAI cache headers in your request to cut repeat-prompt costs on resumed conversations.

A browser client can consume the stream with EventSource only via GET; for POST, use fetch with a ReadableStream reader. The server contract above is stable regardless of client.

You now have a serverless aws lambda response streaming llm endpoint that returns tokens in real time without provisioning a long-lived server. Swap the model name or base URL to change providers without touching handler logic.