LLM API calls from serverless functions fail in ways traditional HTTP clients don’t expect. Building reliable llm api retries aws lambda requires handling provider rate limits, partial responses, and Lambda’s hard timeout model without duplicating expensive generations or hanging the caller.
Step 1: Configure Lambda for deterministic timeouts
Start by treating the Lambda timeout as a hard budget, not a safety net. Set it to the maximum latency your caller can tolerate, then make the HTTP client timeout strictly smaller. A 10-second HTTP timeout inside a 30-second Lambda leaves room for two retries and cold-start overhead before the function dies.
Put the HTTP client at global scope so Lambda reuse keeps connections warm across invocations:
import httpx
client = httpx.Client(timeout=10.0)
def lambda_handler(event, context):
# client is reused on warm starts
...For llm api retries aws lambda, never rely on the default 60-second socket timeout. Many providers hang instead of returning a 504, and a hung socket will burn your entire Lambda window. Also allocate enough memory: CPU scales with memory on Lambda, and a faster JSON parse reduces the chance of a timeout on large responses.
Step 2: Make the completion call explicit and typed
Use the OpenAI-compatible request shape. It works against OpenAI, Azure, or any gateway that exposes /v1/chat/completions, which means you can swap providers without rewriting the retry logic.
import os
import httpx
ENDPOINT = os.environ["LLM_ENDPOINT"]
API_KEY = os.environ["LLM_API_KEY"]
client = httpx.Client(timeout=10.0)
def complete(prompt: str, idem_key: str) -> str:
resp = client.post(
f"{ENDPOINT}/v1/chat/completions",
headers={
"Authorization": f"Bearer {API_KEY}",
"Idempotency-Key": idem_key,
},
json={
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
"seed": 42,
},
)
resp.raise_for_status()
return resp.json()["choices"][0]["message"]["content"]Temperature 0 and a fixed seed make the output deterministic, which is the foundation for safe retries. If you retry a non-deterministic call, you may get a different token stream and corrupt downstream state.
Step 3: Add exponential backoff with jitter
Wrap the call with tenacity. Retry on 429 and 5xx, plus transport errors. Stop after a small number of attempts to stay inside the Lambda budget. With a 10-second timeout, three attempts worst-case is ~30 seconds including jitter, so set the Lambda timeout to 35 seconds.
from tenacity import (
retry,
wait_random_exponential,
stop_after_attempt,
retry_if_exception_type,
)
import httpx
@retry(
retry=retry_if_exception_type((httpx.HTTPStatusError, httpx.TransportError)),
wait=wait_random_exponential(multiplier=1, max=10),
stop=stop_after_attempt(3),
reraise=True,
)
def complete_retry(prompt: str, idem_key: str) -> str:
resp = client.post(
f"{ENDPOINT}/v1/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}", "Idempotency-Key": idem_key},
json={
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
"seed": 42,
},
)
if resp.status_code in (429, 500, 502, 503, 504):
resp.raise_for_status() # triggers retry
resp.raise_for_status()
return resp.json()["choices"][0]["message"]["content"]The random jitter prevents a thundering herd when a provider recovers and every Lambda simultaneously retries. This is the core of any llm api retries aws lambda implementation.
Step 4: Make retries idempotent at the provider level
Retrying a generation that already succeeded but dropped the response wastes tokens and may double-post side effects. Pass an Idempotency-Key and reuse the same seed. If you route through n4n.ai, it honors client routing directives and forwards provider cache-control hints, so a repeated key with an identical payload hits cache instead of re-executing the model.
When the gateway doesn’t support that, store the key in DynamoDB with a short TTL and check before calling:
import boto3, time, os
ddb = boto3.resource("dynamodb").Table(os.environ["IDEM_TABLE"])
def complete_idem(prompt, idem_key):
if ddb.get_item(Key={"key": idem_key}).get("Item"):
return "cached"
ddb.put_item(Item={"key": idem_key, "ttl": int(time.time()) + 600})
return complete_retry(prompt, idem_key)The TTL avoids unbounded table growth. For write-heavy workloads, use a conditional put to handle concurrent retries safely.
Step 5: Push long-tailed retries out of Lambda
If the provider is deeply degraded, three in-Lambda retries won’t save you. Offload to SQS with a delay, and let a second Lambda consume the queue. This pattern for llm api retries aws lambda decouples user latency from provider recovery time.
import boto3, json, os
sqs = boto3.client("sqs")
def lambda_handler(event, context):
try:
return {"result": complete_idem(event["prompt"], event["idem_key"])}
except Exception:
sqs.send_message(
QueueUrl=os.environ["RETRY_QUEUE"],
MessageBody=json.dumps(event),
DelaySeconds=30,
)
return {"status": "queued_for_retry"}The queue consumer re-invokes the same logic with a fresh Lambda context. Set the SQS visibility timeout longer than your function timeout to avoid duplicate deliveries while a retry is still running.
Step 6: Wire fallback to a secondary provider
Hard-coding one model is fragile. Use a gateway that performs automatic fallback when a provider is rate-limited or degraded. If you call an endpoint that addresses 240+ models behind one OpenAI-compatible URL, you can shift the model field or let the gateway route based on health.
def complete_with_fallback(prompt, idem_key):
try:
return complete_idem(prompt, idem_key)
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
# try a smaller model on same gateway
return complete_idem_small(prompt, idem_key)
raiseKeep the fallback model deterministic too. A smaller model may produce different text, so only use fallback when the output is consumed by a system that tolerates variance, not when exact reproduction is required.
Step 7: Verify retries in a staging environment
Deploy the function, then point LLM_ENDPOINT at a mock that returns 429 twice and then 200. Use this stub:
from http.server import BaseHTTPRequestHandler, HTTPServer
class H(BaseHTTPRequestHandler):
count = 0
def do_POST(self):
self.count += 1
if self.count <= 2:
self.send_response(429); self.end_headers(); return
self.send_response(200)
self.end_headers()
self.wfile.write(b'{"choices":[{"message":{"content":"ok"}}]}')
HTTPServer(("0.0.0.0", 8080), H).serve_forever()Invoke the Lambda with a test event. In CloudWatch, confirm two HTTPStatusError log lines followed by a success. If you wired SQS, send one message and watch it requeue once, then complete after the mock recovers.
For unit tests, use responses or pytest-httpx to simulate status codes without network:
import pytest, httpx
from your_module import complete_retry
def test_retry_then_success(httpx_mock):
httpx_mock.add_response(status_code=429)
httpx_mock.add_response(status_code=429)
httpx_mock.add_response(json={"choices":[{"message":{"content":"hi"}}]})
assert complete_retry("prompt", "key") == "hi"A sudden spike in retry metrics in production means the provider is throttling, not that your llm api retries aws lambda code is broken.
Step 8: Clean up and cost-guard
Set a per-invocation token cap. If the prompt plus max_tokens exceeds your budget, fail fast instead of retrying. Retries multiply token cost linearly.
MAX_TOKENS = 2000
if len(prompt) > MAX_TOKENS:
return {"error": "prompt_too_long"}Also add a dead-letter queue on the SQS consumer so permanently failing messages don’t loop forever. With these steps, llm api retries aws lambda become a controlled, observable process rather than a silent loop burning your account.