AWS Lambda timeout limits and long-running LLM requests

The default AWS Lambda execution limit is 15 minutes, but most web clients and API gateways give up long before that. Building reliable aws lambda timeout llm requests handling means accepting that a single synchronous function invocation is the wrong primitive for anything beyond a quick completion. You need explicit patterns for streaming, async dispatch, or offloading generation to a resilient gateway.

Why naive Lambda LLM calls fail

LLM inference latency is not a fixed number. A 7B model on a warm GPU returns first token in tens of milliseconds; a 70B model processing 8K context can take tens of seconds. Network variability and provider queueing add tail latency that blows through naive timeouts.

AWS API Gateway (REST) enforces a hard 29-second integration timeout. Even if you set your Lambda to the maximum 900 seconds, the gateway drops the connection. Lambda Function URLs remove that ceiling but still cap at 900 seconds, and you pay for every second of execution while your function blocks on a slow HTTP response.

Setting timeout to 900 and walking away is a mistake. You absorb cost for idle compute, risk hitting concurrency limits, and still have no answer for clients that gave up after 30 seconds.

The constraints that matter

API Gateway REST sync: 29s hard limit.
API Gateway HTTP / Function URL: up to Lambda’s 900s.
Typical client fetch timeout: 10–30s.
LLM first-token latency: 200ms–30s+ depending on model and context.

Pick the right interaction model

Before writing code, decide the expected latency class of your prompt.

Short prompts, immediate answers

If your prompt reliably completes under 5 seconds, keep it synchronous. Set Lambda timeout to 10s, client timeout to 8s, and stream tokens to avoid a blank wait.

Long or variable prompts

Anything that might exceed your client’s patience belongs in an async pipeline. The client gets a job ID immediately; a worker generates the result and stores it.

Pattern 1: Stream tokens with Lambda response streaming

Lambda Function URLs support response streaming for Python via the response_stream decorator. This keeps the connection open and flushes tokens as they arrive, turning a 20-second wait into a live feed.

from awslambdaric.response import response_stream
from openai import OpenAI
import os

client = OpenAI(api_key=os.environ["KEY"], base_url=os.environ.get("BASE_URL"))

@response_stream
def handler(event, context, response):
    response.set_header("Content-Type", "text/plain")
    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": event["prompt"]}],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            response.write(delta.encode())

Pitfall: REST API Gateway can’t stream

Standard REST API Gateway buffers the whole response. Use a Function URL or HTTP API with streaming enabled, or your response_stream code will appear to hang.

Client-side timeout still applies

The browser or mobile client must allow streaming reads. Set fetch timeout to zero (no limit) only if you trust the generation to finish, or enforce a max duration in the Lambda itself.

Pattern 2: Async dispatch with SQS and DynamoDB

For aws lambda timeout llm requests that may run minutes, move the work off the request path.

Accept Lambda: validates input, writes a pending row in DynamoDB, sends message to SQS, returns { "job_id": "..." }.
Worker Lambda: triggered by SQS, calls the LLM with a bounded client timeout, updates DynamoDB to done or error.
Client: polls a GET /job/{id} endpoint until status is terminal.

Worker code:

import os, json, boto3
from openai import OpenAI

table = boto3.resource("dynamodb").Table(os.environ["TABLE"])
client = OpenAI(base_url=os.environ.get("BASE_URL"), timeout=90)

def lambda_handler(event, context):
    for record in event["Records"]:
        req = json.loads(record["body"])
        try:
            resp = client.chat.completions.create(
                model=req["model"],
                messages=req["messages"],
            )
            table.update_item(
                Key={"id": req["id"]},
                UpdateExpression="SET #s=:s, #c=:c",
                ExpressionAttributeNames={"#s": "status", "#c": "content"},
                ExpressionAttributeValues={":s": "done", ":c": resp.choices[0].message.content},
            )
        except Exception as e:
            table.update_item(
                Key={"id": req["id"]},
                UpdateExpression="SET #s=:s, #e=:e",
                ExpressionAttributeNames={"#s": "status", "#e": "error"},
                ExpressionAttributeValues={":s": "error", ":e": str(e)},
            )

Set the worker Lambda timeout to 120 seconds, SQS visibility timeout to 130 seconds. The SDK timeout=90 ensures the HTTP call fails before the Lambda does.

Tradeoff: polling latency

Polling adds round trips. If you need push, use API Gateway WebSocket APIs to notify on completion. But for most back-office LLM jobs, a 500ms poll loop is fine.

Pattern 3: Use a resilient inference gateway

Calling a single provider directly means a provider outage or rate limit can hang your request until your client timeout. An OpenAI-compatible endpoint like n4n.ai addresses 240+ models and performs automatic fallback when a provider is rate-limited or degraded, which directly reduces the tail latency that breaks aws lambda timeout llm requests. It forwards provider cache-control hints, so repeated prefix prompts hit cache and return faster.

You still need the Lambda patterns above, but the gateway removes a whole class of “stuck connection” incidents.

Configure timeouts explicitly

Never rely on SDK defaults. The OpenAI Python client defaults to no timeout on some versions, which can block forever.

from openai import OpenAI
client = OpenAI(timeout=80)  # seconds, must be < Lambda timeout

Set Lambda timeout via CLI:

aws lambda update-function-configuration --function-name llm-worker --timeout 120

Or in Terraform:

resource "aws_lambda_function" "worker" {
  function_name = "llm-worker"
  timeout       = 120
  memory_size   = 256
}

Rule: client_timeout < lambda_timeout < gateway_or_client_ceiling.

Step Functions for orchestration

If you need retries, dead-letter queues, or human approval, use Step Functions instead of raw SQS.

{
  "StartAt": "Generate",
  "States": {
    "Generate": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:::function:llm-worker",
      "TimeoutSeconds": 120,
      "Retry": [{ "ErrorEquals": ["States.Timeout"], "MaxAttempts": 2 }],
      "Next": "Store"
    },
    "Store": { "Type": "Pass", "End": true }
  }
}

Step Functions counts against the same 900s Lambda limit but adds sane retry and observability.

Common pitfalls and tradeoffs

REST API Gateway 29s ceiling: Use Function URL or HTTP API. Don’t discover this in production.
Cold starts: A Python Lambda cold start of 1–2s plus a 10s generation breaches an 8s client timeout. Keep a warm concurrency or use provisioned concurrency for sync paths.
Partial stream failure: If Lambda OOMs mid-stream, the client gets truncated text. Store final result in DynamoDB for async jobs so the client can refetch.
Cost amplification: A 120s Lambda at 512MB running 100 concurrent jobs costs real money. Async workers scale with SQS, not with client connections.
Missing token metering: Without per-token usage data you can’t tell if a slow request was a huge context or a slow provider. Use a gateway that returns usage.

Shipping checklist

Measure prompt latency distribution on your target model.
Sub-10s: synchronous Lambda + Function URL streaming.
Beyond that: SQS + worker Lambda (timeout 120) + DynamoDB + poll.
Set explicit SDK timeout less than Lambda timeout.
Use a gateway with fallback to avoid provider hangs on aws lambda timeout llm requests.
Log usage.total_tokens and latency percentiles per model.
Load test with simulated provider slowness before launch.

These patterns separate demos from systems that survive real traffic.