Designing idempotent retries for multi-provider LLM calls

Retrying a failed LLM request looks trivial until you account for double charges, duplicated side effects, and provider-specific quirks. Building idempotent retries llm api calls across multiple providers requires explicit request keys, careful fallback logic, and a clear story for partial responses.

1. Generate a stable idempotency key before the first call

The foundation of any idempotent retries llm api design is a key that uniquely identifies a logical operation, not a transport attempt. If you key on the HTTP request ID, every retry is a new operation and you lose all protection.

Generate the key in the calling service, close to the business action that triggered the LLM call. Use a UUIDv4 plus a domain suffix, or a hash of the prompt plus a client-supplied nonce. The key must be deterministic per logical turn but unique enough to avoid cross-user collisions.

import uuid

def make_idempotency_key(user_id: str, conversation_id: str, turn: int) -> str:
    # Stable per logical turn, not per network attempt
    return f"llm:{user_id}:{conversation_id}:{turn}:{uuid.uuid4()}"

Persist this key in your request log before sending. If the process crashes after the LLM returns but before you store the result, the retry will reuse the same key and the provider can return the cached response. Do not derive the key from a timestamp alone; concurrent retries will diverge.

2. Send the key as a first-class header

Most LLM gateways and some providers accept an Idempotency-Key header. Even when the upstream model vendor ignores it, your own proxy or gateway should honor it. For an OpenAI-compatible endpoint, the call is just a POST with an extra header.

curl https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Idempotency-Key: llm:user_42:conv_7:3:9b1c2f3a" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Summarize the ticket"}]
  }'

If you use the Python SDK, wrap it:

from openai import OpenAI

client = OpenAI(base_url="https://api.example.com/v1")

def chat_with_key(key: str, **kwargs):
    return client.chat.completions.create(
        **kwargs,
        headers={"Idempotency-Key": key},
    )

The header must travel with every retry. Do not regenerate it mid-flight. If you switch providers inside a retry loop, the key stays constant; only the destination changes.

3. Abstract providers behind a single retry loop

When you call multiple providers—say OpenAI, Anthropic, and a self-hosted vLLM—you need one retry controller that owns the key and the fallback order. A naive approach retries each provider independently, which multiplies the chance of duplicate side effects.

PROVIDERS = ["openai", "anthropic", "local"]

def call_with_fallback(key, messages):
    last_err = None
    for provider in PROVIDERS:
        try:
            return send_to_provider(provider, key, messages)
        except RateLimitError as e:
            last_err = e
            continue  # try next provider
        except ProviderTimeout as e:
            last_err = e
            # timeout may mean the request is still processing; do not blind-fallback
            raise UncertainFailure(key, provider, e)
    raise AllProvidersFailed(last_err)

A gateway such as n4n.ai offers automatic fallback when a provider is rate-limited or degraded, but the idempotency key must still be supplied by your client to prevent duplicate billing across the fallback chain.

The critical rule: if a call times out, you cannot assume it failed. Mark the attempt as “unknown” and poll with the same key rather than immediately switching providers.

3.1 Track attempt state externally

Use a small KV store (Redis, DynamoDB, or even a SQLite table) to record each key’s status.

{
  "key": "llm:user_42:conv_7:3:9b1c2f3a",
  "status": "processing",
  "provider": "openai",
  "created_at": "2025-04-12T10:22:01Z",
  "attempts": 1
}

On retry, read the record. If status is completed, return the stored completion. If processing and younger than a TTL, block the new attempt or poll. This turns a distributed retry problem into a local state machine.

3.2 Polling for uncertain outcomes

When a timeout occurs, the request may have been processed and the response lost. With the idempotency key stored, you can issue a status query if the gateway supports it.

def poll_key(client, key):
    resp = client.get(f"/v1/idempotency/{key}")
    if resp.status == "completed":
        return resp.completion
    elif resp.status == "processing":
        raise StillProcessing()

If no such endpoint exists, wait a bounded backoff and retry the exact same request; the gateway should return the original result if it succeeded. Never reuse the key with different request parameters—most APIs respond 409 Conflict, which is a bug signal, not a retry signal.

4. Make retries safe for non-idempotent side effects

LLM calls often trigger tool use: writing to a database, sending an email, posting a Slack message. Idempotent retries llm api patterns must extend to those tools. Pass the same idempotency key into the tool invocation so the tool can dedupe.

def send_slack_message(key, channel, text):
    # Slack accepts idempotency via unique_ts or your own dedupe table
    if redis.get(f"slack:{key}"):
        return
    api.chat_postMessage(channel=channel, text=text, metadata={"idempotency_key": key})
    redis.setex(f"slack:{key}", 86400, "1")

If the tool does not support idempotency, wrap it with a claim-check pattern: write a pending job row with the key as primary key, and have the worker reject duplicates. The LLM response itself may be idempotent, but the world it mutates is not.

5. Streaming complicates everything

With streaming responses, the connection can drop after 50 tokens. You cannot simply replay the whole stream; the user already saw partial output. Two strategies:

5.1 Retry only before first token

If you have received zero bytes, treat it like a non-streaming failure and retry with the same key. Most gateways will honor the key and return a fresh stream.

5.2 Resume with offset

Some providers support resume_from or you can cache streamed tokens locally and request completion from token N. This is provider-specific and breaks the uniform abstraction. Prefer the first strategy and design UI to show a “regenerating” state if the pre-first-token window is missed.

async function streamWithRetry(key: string, messages: any[]) {
  for (let attempt = 0; attempt < 3; attempt++) {
    const stream = await client.chat.stream({ key, messages });
    let gotToken = false;
    for await (const chunk of stream) {
      gotToken = true;
      yield chunk;
    }
    if (gotToken) return; // success
    // else loop with same key
  }
  throw new Error("stream failed before first token");
}

6. Common pitfalls and tradeoffs

Pitfall: key collision across logical operations. If you hash only the prompt, two users with the same prompt share a key and one gets the other’s cached answer. Always include a tenant or user scoping element.

Pitfall: TTL too short. Idempotency records must outlive the maximum retry window plus any background reconciliation. 24 hours is a safe default for human-facing features; batch jobs may need longer.

Pitfall: ignoring 409 Conflict. Reusing a key with changed parameters should fail loud. Silently generating a new key hides a client bug and corrupts metering.

Tradeoff: strict dedupe vs. flexibility. If you key on the exact message array, a minor whitespace change forces a new call. That is usually correct—you want semantic changes to bypass the cache. But for prompt-template systems, consider normalizing whitespace and sorting JSON keys before hashing.

Tradeoff: fallback increases tail latency. Checking the KV store on every call adds a round trip. Use a local in-memory cache with async write-through to reduce p99.

Pitfall: ignoring provider cache hints. Some providers honor cache_control to persist prompts for hours. If you send a new idempotency key with the same cached prompt, you may still get a cache hit upstream but double-count tokens in your metering. Forward the same cache directives on retries.

Tradeoff: metering alignment. Even with idempotent retries, your per-token usage metering must dedupe on the key, or you will over-report. The retry controller should emit a single usage record per completed key, not per attempt.

7. A minimal ordered checklist

1. Create the idempotency key at the business layer, scoped to tenant + action.
2. Persist key status (pending) before any network call.
3. Send the key as a header to your gateway or provider.
4. On timeout, mark unknown; poll with key instead of blind fallback.
5. On 5xx or rate limit, advance to next provider with the same key.
6. For tools triggered by the model, propagate the key into the tool call.
7. For streaming, retry only before first token; otherwise surface regeneration UI.
8. Expire keys after a TTL longer than your longest retry horizon.

Following this path makes idempotent retries llm api calls predictable even when you span three providers and a flaky network. The extra plumbing pays for itself the first time a user hits submit twice.