5 patterns for retrying failed steps in async agent chains

Retry patterns async agents need differ from classic synchronous service retries because agent steps often run for minutes, invoke non-idempotent tools, and depend on external LLM providers with volatile latency. A naive while try loop will corrupt state or silently double-charge tokens. The five patterns below are what we ship in production agent chains to keep long-running workflows reliable without babysitting them.

1. Exponential backoff with jitter and capped retries

Transient failures dominate async agent chains: a tool API returns 503, a vector store times out, or the LLM gateway throttles you. Retrying immediately floods the dependency and turns a blip into an outage. Exponential backoff spreads the load, and jitter prevents thundering herds when many agents restart simultaneously after a deploy.

Why cap and jitter

Unbounded backoff can delay a critical step for minutes when a quick fallback would have succeeded. Cap the interval (e.g., 30s) and add random jitter so correlated failures don’t sync up. In Python this is a dozen lines:

import asyncio, random

class TransientError(Exception):
    pass

async def call_with_backoff(fn, max_attempts=5, base=0.5, cap=30.0):
    attempt = 0
    while attempt < max_attempts:
        try:
            return await fn()
        except TransientError:
            if attempt == max_attempts - 1:
                raise
            sleep = min(cap, base * (2 ** attempt)) + random.uniform(0, 0.5)
            await asyncio.sleep(sleep)
            attempt += 1

Wire this around every network call your agent makes. The max_attempts should reflect step criticality: a search query might get 3, a payment capture gets 1 with immediate escalation. Classify errors explicitly—only TransientError (timeouts, 429, 503) should hit this loop. A ValueError from malformed JSON is permanent and will waste attempts.

A subtle bug is catching overly broad exceptions. If you wrap except Exception, a programming error in your step function will be silently retried and buried. Log the failure category and emit a metric so you can see retry pressure building before it becomes a backlog.

2. Idempotent step execution with deduplication keys

Async agents frequently retry steps that already succeeded because the failure happened after the side effect but before the ack. Without idempotency you double-send emails or double-write database rows. The fix is a deduplication key derived from the run ID, step name, and input hash, stored in a fast key-value store.

Implementation sketch

import redis, json, hashlib

r = redis.Redis()

def step_key(run_id, step, payload):
    h = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()[:16]
    return f"{run_id}:{step}:{h}"

async def run_idempotent(run_id, step, payload, coro):
    key = step_key(run_id, step, payload)
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    result = await coro()
    r.set(key, json.dumps(result), ex=86400)
    return result

The key must be stable across retries. If the step calls an external API that mutates state, ensure the API itself accepts an idempotency key header—Stripe and most payment rails do. For LLM-only steps, caching the response is safe because the input hash captures the prompt and model version. Set a TTL longer than the maximum workflow lifetime; otherwise a slow run could recompute a step after cache expiry and duplicate the effect.

Do not use the run ID alone as the key. Two different inputs to the same step would collide and return stale output. Include a content hash of the arguments. If the step reads from a mutable external source (e.g., “current weather”), idempotency within a single run is still correct, but cross-run caching should be disabled.

3. Checkpoint-and-resume with durable state stores

Long-running agent workflows span worker crashes, deploys, and cloud zone blips. If you keep step outputs only in memory, a restart forces a full re-run, wasting tokens and risking duplicate side effects. Persist a checkpoint after each completed step in a durable store like SQLite, Postgres, or S3.

Schema and resume loop

CREATE TABLE step_state (
  run_id     TEXT NOT NULL,
  step_name  TEXT NOT NULL,
  status     TEXT NOT NULL,
  output     TEXT,
  updated_at TIMESTAMP DEFAULT NOW(),
  PRIMARY KEY (run_id, step_name)
);

async def execute_chain(run_id, steps):
    for name, fn in steps:
        row = db.fetch_one(
            "SELECT status, output FROM step_state WHERE run_id=%s AND step_name=%s",
            run_id, name)
        if row and row["status"] == "done":
            continue
        db.execute("INSERT INTO step_state VALUES (%s,%s,'running',NULL,NOW()) "
                   "ON CONFLICT DO UPDATE SET status='running'", run_id, name)
        try:
            out = await fn()
            db.execute("UPDATE step_state SET status='done', output=%s WHERE run_id=%s AND step_name=%s",
                       json.dumps(out), run_id, name)
        except Exception:
            db.execute("UPDATE step_state SET status='failed' WHERE run_id=%s AND step_name=%s",
                       run_id, name)
            raise

On restart, the loop skips finished steps and re-runs only the failed or running ones. Combine this with pattern 2 to make the re-run safe. Add a running timeout: if a step stays in running for more than its SLA, treat it as failed and let a fresh worker pick it up. Use a unique constraint and optimistic locking if you run multiple workers on the same run, otherwise two workers will execute the same step concurrently.

Checkpointing adds write latency to every step, but the cost is trivial compared to recomputing a 20-step research chain. Store only the minimal output needed for downstream steps; raw LLM transcripts can be offloaded to blob storage and referenced by URL.

4. Provider fallback and model degradation

LLM provider errors are a special class of transient failure. Rate limits, regional degradations, and model deprecations will hit you in production. Retry patterns async agents use should distinguish “provider unavailable” from “bad output”. If you route through a gateway such as n4n.ai, the automatic fallback when a provider is rate-limited or degraded means you can skip writing custom multi-client logic for availability—but you still need to handle malformed responses at the application layer. The gateway also provides per-token usage metering and honors client routing directives, so repeated prompts hit cache instead of burning tokens.

Application-level model drop

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")

async def complete_with_fallback(messages):
    try:
        return client.chat.completions.create(
            model="anthropic/claude-3.5-sonnet", messages=messages)
    except (BadResponseError, ValueError):
        # fall back to cheaper model on schema mismatch, not on transport error
        return client.chat.completions.create(
            model="openai/gpt-4o-mini", messages=messages)

The gateway already forwards provider cache-control hints, so a retried identical prompt is served from cache. Keep fallback models semantically close; dropping from a reasoning model to a tiny one will silently degrade agent quality. Never use fallback to mask persistent prompt errors—if the primary model returns a schema violation 100% of the time, fix the prompt, don’t downgrade the model forever.

Transport-level retries (connection reset, 502) should be left to the gateway’s automatic fallback. Application-level retries should target output validation: JSON parse failure, missing required fields, or confidence below threshold. That separation keeps your code clean and lets the infrastructure handle volatility.

5. Dead-letter queues with structured error context

Some steps fail after all retries: a third-party API is down for hours, or the input violates a hard constraint. Pushing these to a dead-letter queue (DLQ) instead of crashing the chain lets the rest of the workflow proceed and gives operators a precise record to act on.

Queue payload shape

{
  "run_id": "run_8f2c",
  "step": "invoice_creation",
  "error_type": "ValueError",
  "attempts": 5,
  "last_error": "missing tax_id for EU vendor",
  "timestamp": "2024-11-12T14:22:01Z",
  "payload": {"vendor": "acme.de", "amount": 4200}
}

Use a real broker (SQS, Redis Streams, or Kafka) so the DLQ survives process death. A separate worker consumes these messages, alerts a human, or triggers a compensating action like a refund hold. The key is structured context: a bare stack trace is useless when you have 10,000 agent runs per hour. Include the logical step name and the input payload so the failure can be reproduced.

Design the DLQ consumer to be idempotent as well. If it writes a compensation record, use the run_id + step as a primary key. Otherwise a consumer crash will duplicate the human alert or the rollback. Set a retention period and a dead-letter depth alert; a silently growing DLQ is a production incident in disguise.

Synthesis

Pick the minimal set that covers your failure modes; over-engineering retries hides real bugs behind endless loops. The retry patterns async agents need are boring infrastructure, not ML cleverness—get them right and the agent looks magical.