A long-running agent that schedules work across hours or days cannot assume a tool executes exactly once. Network timeouts, orchestrator replays, and manual retries turn every external action into an at-least-once delivery problem, which is why building idempotent agent tool calls is the only way to keep state consistent when the same step runs twice. If your agent books a flight, posts a message, or charges a card, a non-idempotent call will eventually fire twice and corrupt the world outside your process.
The at-least-once reality of async agents
Most production agent runtimes are built on top of durable execution engines: Temporal, Celery with retries, AWS Step Functions, or a custom job queue with checkpointing. These systems guarantee at-least-once task execution. They do not guarantee exactly-once. When a worker dies mid-step, the orchestrator reaps the lease and replays the step on another node. From the agent’s perspective, the LLM emitted a tool call, the tool call was issued, and then the process vanished before recording success.
The LLM itself compounds this. In a typical ReAct or plan-and-execute loop, the model generates a tool invocation, the runtime executes it, and then the runtime feeds the result back to the model. If the runtime crashes after the HTTP request but before persisting the observation, the next replay will show the model the same tool call still pending. The model—or the orchestrator’s retry logic—will issue it again.
This is not a corner case. It is the steady-state behavior of any agent that outlives a single HTTP request.
Failure modes without idempotency
Duplicate side effects on external systems
Consider an agent that books travel. A naive tool wrapper looks like this:
def book_flight(origin: str, dest: str, date: str, passenger: str):
resp = requests.post(
"https://api.airline.example/book",
json={"origin": origin, "dest": dest, "date": date, "passenger": passenger},
)
resp.raise_for_status()
return resp.json()If the airline API accepts the booking and returns 201, but the agent’s network connection drops while receiving the response, the local runtime marks the step failed. The orchestrator replays it. The airline has no idea the second request is a duplicate. You now have two seats and two charges.
Double billing and notification storms
Payment APIs are the canonical example. Stripe, PayPal, and most bank rails accept an Idempotency-Key header precisely because they live in the same at-least-once world. If you do not supply one, a retried POST /charges creates a second charge. The same applies to sending transactional email, posting to Slack, or triggering a webhook to a partner.
# Non-idempotent: replaying this curl twice charges twice
curl -X POST https://api.example.com/charge \
-d '{"amount": 4200, "currency": "usd", "customer": "cus_123"}'What idempotent agent tool calls require
An idempotent agent tool call satisfies one rule: executing it N times with the same inputs and the same idempotency key produces the same external effect as executing it once. That requires two mechanisms working together.
- Key propagation – the agent attaches a stable identifier to the call.
- Effect deduplication – either the downstream service honors the key, or the agent maintains a journal that short-circuits repeats.
Key generation strategies
The key must be unique per logical action but stable across retries of that action. Common patterns:
run_id:step_index– deterministic if the agent plan is fixed.run_id:tool_name:hash(args)– safe when the same step could theoretically re-invoke with different args (it shouldn’t, but plans drift).- A random UUID stored in the agent’s state before the first attempt – survives replays because the state is recovered from the journal.
def make_idem_key(run_id: str, step: int, tool: str, args: dict) -> str:
import hashlib, json
args_hash = hashlib.sha256(json.dumps(args, sort_keys=True).encode()).hexdigest()[:16]
return f"{run_id}:{step}:{tool}:{args_hash}"Downstream support vs. local journal
If the tool’s API accepts an idempotency key, use it directly:
requests.post(
"https://api.example.com/charge",
json={"amount": 4200, "currency": "usd", "customer": "cus_123"},
headers={"Idempotency-Key": idem_key},
)If it does not, the agent must emulate idempotency with a persistent store.
class IdempotentToolRunner:
def __init__(self, store):
self.store = store # Redis or DB with TTL
def run(self, key: str, fn):
cached = self.store.get(key)
if cached and cached["status"] == "done":
return cached["result"]
if cached and cached["status"] == "pending":
raise RuntimeError(f"Concurrent or unfinished execution for {key}")
self.store.put(key, {"status": "pending"}, ttl=86400)
try:
result = fn()
self.store.put(key, {"status": "done", "result": result}, ttl=86400)
return result
except Exception:
self.store.delete(key)
raiseImplementing a robust idempotency layer
Persistent store with TTL
The journal must outlive the agent’s maximum possible runtime. For an agent that can be suspended for human approval for a week, a 7-day TTL is the minimum. Use a store that survives process restarts: Redis, DynamoDB, Postgres. Do not use in-memory maps.
Crash before journal write
The tricky window is between a successful external call and recording the result. If the call succeeded but the store update failed, a replay sees pending and refuses to re-execute, leaving the agent stuck. Mitigate by making the external call itself queryable: many payment systems let you retrieve a charge by idempotency key. For custom tools, design a GET /status?key=... endpoint.
{
"idempotency_key": "run_99:3:charge:ab12cd34",
"status": "completed",
"result": {"charge_id": "ch_456", "amount": 4200}
}Reconciliation on partial success
If the downstream service is itself eventually consistent, treat the journal as the source of truth for the agent but trigger a background reconciliation job that verifies the external state matches. This is standard in financial pipelines and applies directly to agents that move money or inventory.
Tradeoffs: when idempotency is overkill
Idempotency is not free. It adds a storage write per tool call, forces you to design key schemes, and complicates error handling. Be honest about where it matters.
- Read-only calls (
GETthat does not mutate) are naturally idempotent. Cache them, but don’t build journals. - Fixed-state writes (
SET user_status = 'approved') are idempotent by nature; replaying just re-sets the same value. - Low-stakes internal telemetry can tolerate duplicates; a doubled log line is annoying, not catastrophic.
Anything that touches money, sends external notifications, allocates scarce resources (seats, instances, licenses), or mutates another company’s database must be wrapped as idempotent agent tool calls. The blast radius of a duplicate is unbounded; the cost of the key is a few bytes.
Inference gateway note
An inference gateway like n4n.ai can mask provider degradation with automatic fallback across 240+ models, but that only covers the LLM completion step. Your agent’s tool execution layer still lives in at-least-once land; fallback at the model layer does not make a non-idempotent POST idempotent. Route directives and cache-control hints help with token cost and latency, not with duplicate side effects downstream.
Decisive takeaway
If your agent runs longer than a single synchronous request/response cycle, treat every state-mutating tool as replayable by default. Generate a stable idempotency key per logical step, propagate it to any downstream service that supports it, and maintain a persistent journal for those that don’t. The engineering tax is fixed and small. The cost of a duplicated charge, duplicated booking, or duplicated public post is a customer-trust incident. Build idempotent agent tool calls now, because the replay is not a question of if—it is a question of when.