Rate limiting LLM calls from AWS Lambda with SQS queues

Throttling outbound requests to a language model from ephemeral compute is a solved problem only if you design for it. This guide shows how to implement aws lambda sqs rate limiting llm inference so your functions never blow past provider quotas or burn cash on 429s.

Step 1: Create an SQS queue sized for your LLM latency

SQS is the right buffer because Lambda scales horizontally and will happily launch hundreds of concurrent executions. If each execution calls an LLM with a 10–30 second response time, you need a queue that hides messages long enough to avoid duplicate processing.

The core of aws lambda sqs rate limiting llm design is the visibility timeout. Set it longer than your function’s timeout plus any retry overhead.

Use the AWS CLI to create a standard queue:

aws sqs create-queue \
  --queue-name llm-inference-queue \
  --attributes VisibilityTimeout=60,MessageRetentionPeriod=86400

VisibilityTimeout=60 means a message fetched by Lambda stays invisible for 60 seconds. If the function finishes and deletes it, great. If the function dies or times out, the message reappears for another consumer. Message retention of one day covers backpressure scenarios.

Producers can be anything: an API Lambda, a Step Functions state, or a cron job. They only need sqs:SendMessage permission.

Step 2: Cap Lambda concurrency to match provider limits

Unbounded concurrency is the fastest way to hit a provider’s requests-per-minute ceiling. Reserve concurrency on the consumer function to a number your LLM plan can sustain.

Do the math. If your provider allows 50 RPM and each call takes ~2 seconds, one execution occupies a slot for 2 seconds, so a single execution can do 30 calls per minute. Twenty concurrent executions yield 600 calls/minute of capacity—well above 50. So the safe concurrency is actually RPM / (60 / avg_latency_sec). For 50 RPM and 2s latency: 50 / 30 = 1.66 → 2 concurrent executions. Most providers have higher limits; adjust accordingly.

aws lambda put-function-concurrency \
  --function-name llm-consumer \
  --reserved-concurrent-executions 20

Reserved concurrency guarantees a hard ceiling enforced by the Lambda control plane. It also reserves that capacity exclusively, so other functions in your account cannot steal it. When tuning aws lambda sqs rate limiting llm throughput, treat this number as your primary dial.

Step 3: Write the consumer handler

The handler pulls a batch from SQS (event source mapping delivers up to 10 messages). For each message, parse the prompt, call the LLM, and persist the result. Use an HTTP client with a timeout. Below is a minimal Python example using the OpenAI SDK pointed at an OpenAI-compatible gateway.

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1")
)

def handler(event, context):
    failures = []
    for record in event["Records"]:
        msg_id = record["messageId"]
        try:
            payload = json.loads(record["body"])
            resp = client.chat.completions.create(
                model=payload["model"],
                messages=payload["messages"],
                timeout=25
            )
            # Persist resp.choices[0].message to DynamoDB or S3
        except Exception as e:
            # Returning the message ID in batchItemFailures triggers SQS redelivery
            failures.append({"itemIdentifier": msg_id})
    return {"batchItemFailures": failures}

If you route through n4n.ai, its automatic fallback covers upstream provider degradation, but your own concurrency cap still prevents self-inflicted throttling. The batchItemFailures pattern is critical: it lets SQS re-deliver only the messages that actually failed instead of the whole batch.

Handle rate-limit exceptions explicitly if you want finer control. Catch openai.RateLimitError and append to failures; the visibility timeout will retry later.

Step 4: Configure the dead-letter queue and redrive

Transient 429s should be retried via visibility timeout, but poison messages (malformed JSON, permanent auth failure) need a DLQ. Create a DLQ and set a redrive policy on the main queue.

aws sqs create-queue --queue-name llm-dlq
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url $DLQ_URL \
  --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

aws sqs set-queue-attributes \
  --queue-url $MAIN_URL \
  --attributes "{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":5}\"}"

After five failed receives, a message lands in llm-dlq for offline inspection. Wire a CloudWatch alarm:

aws cloudwatch put-metric-alarm \
  --alarm-name llm-dlq-nonzero \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=llm-dlq \
  --threshold 0 --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 --period 300 --statistic Maximum

Step 5: Tune the event source mapping for batching

Default SQS->Lambda mapping invokes with whatever is available. To increase efficiency, set BatchSize and MaximumBatchingWindowInSeconds. Batching reduces invocation overhead but does not change concurrency cap.

aws lambda create-event-source-mapping \
  --function-name llm-consumer \
  --event-source-arn $MAIN_QUEUE_ARN \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5 \
  --function-response-types ReportBatchItemFailures

The ReportBatchItemFailures flag enables the partial-batch response we used in Step 3. Without it, a single exception causes the whole batch to be redelivered, multiplying load. When tuning aws lambda sqs rate limiting llm workloads, keep batch size modest; large batches increase the blast radius of a timeout.

Step 6: Verify the rate limit holds under load

Verification is practical, not theoretical. Generate a few hundred messages and watch concurrency.

for i in {1..500}; do
  aws sqs send-message \
    --queue-url $MAIN_URL \
    --message-body "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"user\",\"content\":\"ping $i\"}]}"
done

In CloudWatch, check ConcurrentExecutions for llm-consumer. It should plateau at your reserved number (20 here). Check provider metrics or gateway logs for HTTP 429s; there should be none if the cap is correct. If you see throttling, lower reserved concurrency or increase visibility timeout to absorb retries.

Also confirm DLQ stays at zero during the test unless you intentionally send a bad message. Finally, measure end-to-end latency: queue depth should drain steadily, not oscillate.

Why this pattern beats in-function sleep loops

Engineers sometimes try to rate limit by sleeping inside Lambda or using global token buckets in Redis. That wastes billed time and still races under cold starts. SQS decouples ingestion from execution and lets Lambda scale to zero when the queue is empty. The concurrency cap is enforced by the Lambda control plane, not your code, so it cannot be bypassed by a bug.

Operational notes

• Set Lambda timeout to less than the SQS visibility timeout (e.g., 30s timeout, 60s visibility) to avoid double processing.
• Use idempotency keys on LLM requests if the provider supports them; SQS at-least-once delivery means you will process some messages twice.
• If you need per-tenant fairness, use separate queues or message group IDs with FIFO queues.
• Monitor ApproximateAgeOfOldestMessage to detect when your concurrency cap is too low for the incoming rate.
• The aws lambda sqs rate limiting llm approach is boring on purpose: queues, a hard cap, and partial failures. It survives provider outages and your own traffic spikes.