Idempotency keys for LLM API requests: why they matter

A retry after a network timeout should never bill you twice for the same LLM completion or emit two different answers into your database. Idempotency keys llm api solve this by tagging each logical request with a client-generated token that the server uses to deduplicate repeats. Without them, any nontrivial LLM integration will eventually corrupt state during transient failures.

Why LLM requests are not idempotent by default

HTTP GET is safe and idempotent; POST /v1/chat/completions is neither. The model inference call has two failure modes that bite engineers:

1. The request never reached the provider (connection reset, DNS hiccup). The client retries, and a new completion is generated. No harm beyond wasted latency—unless the first one actually did land.
2. The request landed, the provider computed the completion, but the response dropped on the way back (timeout at 30s, load balancer 502). The client has no idea the work happened. A retry creates a second completion, bills tokens again, and may write a contradictory record to your datastore.

LLM outputs are nondeterministic even with temperature: 0 across providers and versions. So the second completion is not even byte-identical. If your system uses the completion to send an email, create a ticket, or update a ledger, you now have duplicate side effects.

What an idempotency key actually buys you

An idempotency key is a client-supplied identifier for a single logical operation. The server (or gateway) records the key alongside the first response and, on subsequent requests with the same key, returns the stored response instead of recomputing. The contract:

• Same key + same request body → same response (replay).
• Different key → treated as new operation.
• Key absent → no guarantees; caller accepts retry risk.

This shifts retry safety from “hope the network is reliable” to “the API contract absorbs duplicates.” For LLM workloads where per-token metering is standard, that directly protects your budget and your data integrity.

Implementing idempotency keys llm api: an ordered path

Follow this sequence when adding deduplication to an LLM call path.

1. Generate the key at the retry boundary

The key must represent the intent, not the transport. Generate it before the first attempt, store it on the unit of work (job row, Kafka message, React query key), and reuse it for all retries of that unit.

import uuid

def new_idempotency_key(work_item_id: str) -> str:
    # Namespace to avoid collisions across services
    return f"llm:{work_item_id}:{uuid.uuid4().hex}"

Do not use a random UUID per HTTP attempt—that defeats the purpose.

2. Transmit the key on every attempt

Most OpenAI-compatible gateways accept an Idempotency-Key header. If you are calling a raw provider that lacks support, you must front it with your own proxy or use a gateway that honors client routing directives.

import requests

def chat_with_idem(prompt: str, idem_key: str):
    return requests.post(
        "https://gateway.example/v1/chat/completions",
        headers={
            "Authorization": "Bearer sk-...",
            "Idempotency-Key": idem_key,
        },
        json={"model": "anthropic/claude-3.5-sonnet", "messages": [{"role": "user", "content": prompt}]},
        timeout=45,
    )

3. Store the first response atomically

Server side, use a datastore with TTL and atomic insert. Redis SET NX is the right primitive; it prevents two concurrent retries from both computing.

import redis, json
r = redis.Redis()

def handle_request(body, idem_key):
    if idem_key:
        cached = r.get(f"idem:{idem_key}")
        if cached:
            return json.loads(cached)
        # Lock to prevent race on concurrent duplicates
        if not r.setnx(f"lock:{idem_key}", "1"):
            # Another worker is computing; block or return 409
            raise Conflict("duplicate in flight")
        r.expire(f"lock:{idem_key}", 30)
    result = compute_completion(body)  # expensive LLM call
    if idem_key:
        r.setex(f"idem:{idem_key}", 86400, json.dumps(result))
        r.delete(f"lock:{idem_key}")
    return result

4. Define replay semantics for status codes

A 200 should replay exactly. A 429 or 500 should not be cached—those are failures, not results. Only persist 2xx (and perhaps 422 validation errors if you want to avoid re-validating). Document this clearly in your internal API spec.

5. Handle streaming separately

Streaming completions break simple replay: you cannot easily re-emit a terminated SSE stream from cache without buffering the whole thing. Options:

• Buffer then replay: Store the full concatenated text; on replay, fake a stream by chunking the stored string. Works but adds latency on cache hit.
• Key only non-streaming calls: Use idempotency for batch/agent steps where streaming is unnecessary; let interactive endpoints accept retry risk.

Code example: minimal Flask gateway with Redis

Below is a stripped-down server that honors Idempotency-Key for a chat endpoint. It does not call a real model; it stubs the expensive part.

from flask import Flask, request, jsonify
import redis, json, time

app = Flask(__name__)
r = redis.Redis(host="localhost", port=6379, db=0)

@app.route("/v1/chat/completions", methods=["POST"])
def completions():
    key = request.headers.get("Idempotency-Key")
    if key:
        hit = r.get(f"idem:{key}")
        if hit:
            return jsonify(json.loads(hit))
        if not r.setnx(f"lock:{key}", "1"):
            return jsonify({"error": "duplicate request in flight"}), 409
        r.expire(f"lock:{key}", 30)

    # Pretend this is a provider call
    time.sleep(0.2)
    resp = {"id": "chatcmpl-123", "choices": [{"message": {"role": "assistant", "content": "ok"}}]}

    if key:
        r.setex(f"idem:{key}", 86400, json.dumps(resp))
        r.delete(f"lock:{key}")
    return jsonify(resp)

if __name__ == "__main__":
    app.run(port=8080)

Client retry logic wraps chat_with_idem in a loop with exponential backoff, always passing the same idem_key.

Common pitfalls and tradeoffs

TTL too short. If your retry window is 24h (e.g., a job queue with delayed retries), a 1h TTL lets a duplicate slip through later. Set TTL to the maximum plausible retry delay plus margin.

TTL too long. Idempotency records accumulate. A 30-day TTL on millions of keys wastes memory. Use a sensible business window—often 24–72h for interactive, 7 days for async jobs.

Key scope leakage. Never reuse a key across different prompts or model versions if you expect different outputs. A key should map to a fixed (request body, model, params) tuple. If the user edits the prompt, issue a new key.

Gateway fallback double-spend. If you route through a gateway such as n4n.ai that performs automatic fallback when a provider is rate-limited or degraded, the idempotency key must be forwarded to the downstream provider or honored at the gateway layer; otherwise a retry that triggers a different backend model will still generate a second completion and double-meter tokens under per-token usage metering. Design your key namespace to survive provider switches.

Concurrent writes. Without the SET NX lock, two retries arriving simultaneously can both compute. The lock adds a small window of 409s, which the client must handle by backing off.

Non-deterministic caching confusion. Storing one completion under a key does not make the model deterministic. It only ensures that caller gets the same answer on retry. Do not assume the key gives you cross-caller reproducibility.

When not to use idempotency keys

Skip them for purely exploratory, non-side-effecting playground calls where the user is typing and explicitly hitting “regenerate.” There, duplicate generations are desired. Also avoid them on websocket token streams where the protocol already manages sequence ids and the client treats drops as UI glitches, not duplicated state.

Deployment checklist

• Key generated once per logical work item, not per HTTP attempt.
• Header Idempotency-Key sent on all retries.
• Server uses atomic SET NX before expensive call.
• Only 2xx responses persisted; errors not cached.
• TTL aligned to max retry delay.
• Streaming endpoints either buffer or exempted.
• Load test concurrent duplicate arrivals (send same key twice in parallel).

Idempotency keys llm api are not optional for production agents that write to external systems. Implement them at the boundary where you first decide to call a model, and treat the key as part of your data model rather than an afterthought header.