Why idempotency keys matter for LLM API retries

LLM inference requests fail often enough that every production client needs a retry path, but a naive retry loop turns a transient timeout into a double-charged duplicate completion. Idempotency keys llm api retries are the standard mechanism to make repeated attempts safe: the client stamps a unique key on the request, and the server promises to run the operation at most once. Without that contract, you are guessing whether a 504 meant the model ran or not.

The failure modes you are actually dealing with

LLM endpoints are slow relative to typical web APIs. A completion can take 2 seconds or 30 seconds depending on output length and load. During that window, any of these happen in real systems:

• The provider returns HTTP 429 because you hit a per-minute token quota.
• The provider returns 503 while a replica restarts.
• A network middlebox drops the connection after the server has already generated the response.
• Your client times out at 10s, but the model finishes at 12s.

In the last two cases, the work was done. If you retry blindly, you incur a second generation, a second token charge, and possibly a contradictory answer stored in your database. Idempotency keys llm api retries exist precisely to close that ambiguity.

What an idempotency key guarantees (and what it doesn’t)

An idempotency key is an opaque string you send in a request header (conventionally Idempotency-Key). The server records the key and the response body the first time it sees it. On any subsequent request with the same key, the server returns the stored response without re-executing the model call.

Guarantees:

• No duplicate side effects on the server (billing, logging, webhooks).
• Safe client retries across crashes, network errors, and timeouts.

Non-guarantees:

• It does not force the model to produce identical text on retry. If the first call succeeded and was stored, you get that first text even if temperature was 0.7.
• It does not validate that the retried request body matches the original. Most servers reject a key reused with a different body via HTTP 409; some ignore the body entirely.
• It does not survive forever. Servers typically keep keys for 24 hours.

Understanding that distinction prevents a class of bugs where engineers expect “idempotent” to mean “fresh but safe.”

Generating keys that survive crashes

The key must be unique per logical operation, not per HTTP attempt. If your worker process dies after sending the request but before persisting the result, a fresh process must be able to reconstruct the same key to retry safely.

Random UUIDv4 works for single-process retries, but it is lost on crash. Prefer a deterministic key derived from your domain:

def make_idempotency_key(tenant_id: str, job_id: str, step: str) -> str:
    # step might be "summary" or "translate"
    return f"{tenant_id}:{job_id}:{step}"

If job_id is a row in your task table, you can always recompute the key. Avoid embedding timestamps or random suffixes in the key itself.

Attaching the key to an OpenAI-compatible request

Most LLM gateways and OpenAI itself accept the header on POST /v1/chat/completions. Using requests:

import requests

API_KEY = "sk-..."

def chat_completion(messages, idempotency_key, model="gpt-4o-mini"):
    return requests.post(
        "https://api.openai.com/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Idempotency-Key": idempotency_key,
            "Content-Type": "application/json",
        },
        json={"model": model, "messages": messages},
        timeout=45,
    )

If you use the official SDK, pass it via extra_headers:

from openai import OpenAI
client = OpenAI()
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize: ..."}],
    extra_headers={"Idempotency-Key": "tenant_1:job_99:summary"},
)

Building a retry loop that reuses the key

The key must stay constant across all attempts. Generate it once outside the loop. Only retry on errors where the server may not have processed the request.

import time
import random
import requests

def complete_with_retry(messages, idempotency_key, max_attempts=4):
    last_err = None
    for attempt in range(max_attempts):
        try:
            resp = chat_completion(messages, idempotency_key)
            if resp.status_code in (429, 500, 502, 503, 504):
                # Safe to retry: request was not completed
                sleep = (2 ** attempt) + random.uniform(0, 0.5)
                time.sleep(sleep)
                continue
            if resp.status_code == 409:
                # Key reused with conflicting body - programming error
                raise ValueError("Idempotency key conflict; check request body")
            resp.raise_for_status()
            return resp.json()
        except requests.RequestException as e:
            last_err = e
            time.sleep(2 ** attempt)
    raise RuntimeError(f"Retries exhausted: {last_err}")

Note the 409 handling: that is the signal you mutated the prompt but kept the key. Fail loud.

Gateway considerations and fallback

When you call a model through an aggregation layer, the key must be forwarded unchanged to the upstream provider. At n4n.ai, the OpenAI-compatible endpoint preserves the Idempotency-Key header across automatic fallback attempts when a provider is rate-limited or degraded, so a retried request can hit a different model backend without creating a second bill. If the gateway silently dropped the key during fallback, your retry would execute twice on the second provider.

Also confirm the gateway forwards provider cache-control hints if you rely on prompt caching; cached completions and idempotency keys interact because a cache hit is effectively a free replay.

Common pitfalls and tradeoffs

Reusing a key for different prompts. Never. A key is a promise about a specific request body. Reusing it across user inputs returns the wrong completion silently.

Assuming idempotency yields fresh output. If the first attempt succeeded and was stored, subsequent retries return the stored text. For creative generation with temperature > 0, that is usually fine—you wanted exactly-once execution. If you need a new sample, use a new key only after confirming the first truly failed.

Retention window mismatch. If your job queue delays a retry beyond the server’s key TTL (commonly 24h), the safety net is gone. Set your queue visibility timeout accordingly.

Conflict errors in distributed systems. Two workers processing the same job ID will send the same key with the same body—that is correct and the second gets the stored result. But if one worker mutates the body, you get a 409. Make job processing singular or include a worker epoch in the key.

Storage and latency cost on the server. Maintaining a key→response store is not free. High-throughput systems should expect the gateway to enforce a TTL and maybe a max key length. Keep keys under 256 bytes.

When not to bother

For a local llama.cpp instance serving a dev laptop, skip the overhead. For any paid hosted inference, idempotency keys llm api retries are mandatory. The cost of one duplicate 100k-token RAG completion dwarfs the trivial client-side effort.

Actionable implementation path

1. Define the key schema. Use tenant_id:job_id:step or similar from your persistent task model.
2. Generate once per job step. Store it in the task row before the first call.
3. Attach on every HTTP attempt. Header Idempotency-Key, same value.
4. Retry only on 429, 5xx, and network exceptions. Never on 4xx except 409 (which is a bug).
5. Handle 409 as a hard failure and alert; it means body/key mismatch.
6. Verify gateway forwarding. If you use a router, confirm it passes the header during provider fallback.
7. Monitor duplicate token spend. If idempotency is working, retried requests should not increase billed tokens after the first success.

Following that ordered path turns a fragile “hope it doesn’t double-charge” loop into a system that survives provider flakiness by design. Idempotency keys llm api retries are not an advanced feature—they are the baseline for billing-correct LLM applications.