Idempotency for LLM API requests means that submitting the same request multiple times with a client-supplied idempotency key produces the same logical outcome as a single submission, without creating duplicate side effects such as double token billing or parallel generations. An idempotent idempotency llm api requests pattern lets you retry a failed POST to a model endpoint safely, because the gateway or provider deduplicates based on that key rather than executing the inference twice.
REST idempotency in one paragraph
HTTP defines GET, PUT, and DELETE as idempotent: repeating them doesn’t change server state beyond the first application. POST is not idempotent by default—each call to /v1/chat/completions triggers a new inference. For idempotency llm api requests, the client attaches a unique Idempotency-Key header; the server records the key alongside the request payload and the returned response, then short-circuits any later call bearing the same key and payload.
How idempotency works for LLM gateways
Key generation and header
The client mints a random string (usually a UUID v4) and sends it in the Idempotency-Key header on every retry of the same logical operation. The key must be unique per logical task, not per HTTP attempt.
curl https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Idempotency-Key: 9f1c2b3a-4d5e-6f7a-8b9c-0d1e2f3a4b5c" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello"}]}'Request fingerprinting
A correct implementation hashes the request body (or canonicalized JSON) and stores a mapping: key -> (request_hash, response). If a second request arrives with the same key but a different body, the gateway rejects it with 422 because the key is being reused inconsistently.
import hashlib, json
def fingerprint(body: dict) -> str:
canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(canonical.encode()).hexdigest()Response replay and TTL
The stored response is replayed for subsequent calls. Keys typically expire after 24 hours; LLM providers don’t keep your completion forever. After expiry, the key is forgotten and a new request executes normally. The client should treat the key as valid only within the retry window of the task.
Streaming considerations
When you set "stream": true, the gateway must buffer the full server-sent event stream before it can replay it deterministically. Some OpenAI-compatible servers forbid idempotency keys on streaming routes; others store the aggregated SSE frames and replay them verbatim. If you need both retries and streaming, confirm the gateway’s behavior before relying on it.
Why idempotency matters for LLM workloads
Token metering is per execution
Most gateways apply per-token usage metering. Without idempotency, a timeout after the model has already generated tokens forces you to either accept a duplicate charge on retry or build fragile custom dedupe logic. With an idempotency key, the retry returns the original response and the original token count—no double meter.
Retries are inevitable
In distributed systems, transient 503s from a provider are common. A naive retry loop on a POST will spin up multiple completions, each consuming context window and money. Idempotent idempotency llm api requests convert that risk into a safe replay.
Failover across providers
Some gateways, including n4n.ai, implement automatic fallback when a provider is rate-limited or degraded; combining that with idempotency keys prevents a retried request from generating a second answer after the fallback occurs. The key travels with the request, so the secondary provider recognizes the duplicate and serves the cached result.
Serverless and queue workers
In a Lambda or queue consumer, a worker may be killed mid-flight. If the job is requeued, a new worker must reuse the same idempotency key to avoid a second generation. Persist the key in the job metadata, not in process memory.
Concrete example: safe retry of a chat call
Suppose your first call times out at 30s. You catch the exception and retry with the same key.
import requests, uuid
def complete(messages, idem_key=None):
idem_key = idem_key or str(uuid.uuid4())
resp = requests.post(
"https://api.example.com/v1/chat/completions",
headers={
"Authorization": "Bearer " + TOKEN,
"Idempotency-Key": idem_key,
"Content-Type": "application/json",
},
json={"model": "claude-3-5-sonnet", "messages": messages},
timeout=30,
)
return resp.json(), idem_key
try:
data, key = complete([{"role": "user", "content": "Summarize RFC 7231"}])
except requests.Timeout:
data, _ = complete(
[{"role": "user", "content": "Summarize RFC 7231"}],
idem_key=key, # reuse
)The second call returns the exact same data object the first call would have returned had it succeeded—including the usage block.
{
"id": "chatcmpl-abc",
"object": "chat.completion",
"usage": {"prompt_tokens": 12, "completion_tokens": 34, "total_tokens": 46},
"choices": [{"message": {"role": "assistant", "content": "RFC 7231 defines HTTP/1.1 semantics..."}}]
}Common misconceptions
“Idempotency means identical model output”
False. LLM sampling is non-deterministic unless temperature is zero and no random seed variance exists. Idempotency at the API layer replays the first response; it does not force the model to regenerate the same text. If the first call succeeded, later calls get that stored text. If the first call never completed, the first successful execution wins, and subsequent retries get its output.
“All POST requests are non-idempotent, so LLM APIs can’t be”
REST theory says POST is not inherently idempotent, but a service can make a POST idempotent by contract. OpenAI-compatible endpoints explicitly support Idempotency-Key on POST. The verb is irrelevant once the server commits to the dedupe contract.
“The server must store the key forever”
Storage is bounded. Typical TTL is 24h. If you retry after expiry, you get a fresh execution. Design your retry window accordingly.
“Idempotency is only for payments”
Stripe popularized the pattern for charges, but any expensive, side-effecting POST benefits—including token-generating inference.
“The key must be kept secret”
The idempotency key is not a security credential. It is an operational dedupe token. Leaking it may let someone replay your exact response, but it cannot authorize new spend beyond the original request scope.
Implementing idempotency in your client
Generate the key at the start of a logical operation, not per HTTP attempt:
async function chatWithRetry(messages: any[]) {
const idemKey = crypto.randomUUID();
for (let i = 0; i < 3; i++) {
try {
const r = await fetch("https://api.example.com/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": `Bearer ${token}`,
"Idempotency-Key": idemKey,
"Content-Type": "application/json"
},
body: JSON.stringify({ model: "gpt-4o-mini", messages })
});
if (r.status === 503) continue;
return await r.json();
} catch (e) {
if (i === 2) throw e;
}
}
}Persist the key in your job queue so that a worker crash and restart can resume the same logical call without duplicating work. Never reuse a key across semantically different prompts; that triggers a 422 or silently returns the wrong cached answer.
Gateway features that complement idempotency
Idempotency handles retry safety, but it does not reduce the cost of the first call. Provider prefix caching (signaled via cache-control hints) and explicit client routing directives reduce redundant compute when the same prefix is sent repeatedly. A gateway that forwards those hints untouched lets you stack semantic caching under the idempotency layer: the first request hits cache or computes, and retries never reach the model.
Checklist for engineers
- Mint one idempotency key per logical inference task; never reuse across distinct prompts.
- Send the same JSON body on retries; gateways reject body mismatches.
- Treat 409/422 on key reuse as a signal of a bug, not a transient error.
- Set client timeouts shorter than the key TTL.
- Log the key alongside the task ID for post-mortems.
- Verify streaming support before combining
stream: truewith idempotency.
Idempotency for idempotency llm api requests is a contract, not a library call. Get the key semantics right and your retry logic becomes boring—which is exactly what you want at 3 a.m. when a provider is flapping.