Async agent orchestration fails in production for the same reasons distributed systems fail: missing failure isolation, no durable state, and optimistic assumptions about latency. Treating an LLM agent as a simple async function invites silent data loss when a worker restarts mid-tool-call. The fix is to borrow battle-tested patterns from service-oriented architectures rather than reinventing a weaker async abstraction.
Why LLM agents are distributed systems problems
An agent that plans, calls tools, and reflects is not a pure function. It issues outbound HTTP requests to vector databases, CRUD APIs, and model endpoints that each have their own failure modes. A single run can span minutes, crossing dozens of I/O boundaries with variable latency. The model output is non-deterministic, so the exact sequence of steps is unknown at compile time.
That combination—long duration, external dependencies, partial failures—is the definition of a distributed workflow. If you would not trust an in-memory asyncio.Task to coordinate a payment and an inventory update, you should not trust it to coordinate autonomous agent steps.
The naive pattern and where it breaks
The first instinct is to write a linear async routine:
async def run_agent(task):
plan = await llm.generate(task)
for step in plan.steps:
result = await call_tool(step)
await llm.feed(result)
return await llm.finalize()This works in a demo. In production, the host recycles, the event loop crashes on an unhandled TimeoutError, or the process is scaled down by the orchestrator. The task vanishes. There is no record of which tool calls completed, so replay either duplicates side effects or skips them.
Worse, because the LLM call itself is slow, you are holding a scarce worker slot for minutes while doing nothing but waiting on network. That is a denial-of-service vector against your own fleet.
Durable execution: the core primitive
The minimal solution is to persist job state before and after every meaningful transition. A relational table or a durable log replaces process memory as the source of truth.
# Schema: jobs(id, payload, state, result, updated_at)
async def worker(poll_interval=1.0):
while True:
job = await db.fetch_one(
"SELECT * FROM jobs WHERE state='queued' ORDER BY id LIMIT 1"
)
if not job:
await asyncio.sleep(poll_interval)
continue
await db.execute(
"UPDATE jobs SET state='running' WHERE id=$1", job["id"]
)
try:
result = await execute_agent(job["payload"])
await db.execute(
"UPDATE jobs SET state='done', result=$1 WHERE id=$2",
result, job["id"]
)
except RetryableError:
await db.execute(
"UPDATE jobs SET state='queued' WHERE id=$1", job["id"]
)
except Exception:
await db.execute(
"UPDATE jobs SET state='failed' WHERE id=$1", job["id"]
)Now a crash mid-execution leaves a running job. A watchdog can reset running jobs older than a threshold back to queued. The work is not lost.
Idempotency keys and recovery
Durable state is necessary but not sufficient. Tool calls—sending an email, posting to Slack—must not fire twice on retry. Generate an idempotency key per step and checkpoint it.
async def execute_step(step, run_id):
key = f"{run_id}:{step.index}"
if await kv.exists(key):
return await kv.get(key)
output = await dispatch_tool(step)
await kv.set(key, output, expire=86400)
return outputThis turns retries into no-ops for completed steps. The agent can resume from the last unchecked point instead of replaying the whole chain.
State machines over free-form chains
Free-form loops inside an LLM prompt are hard to observe and impossible to constrain. Define an explicit finite state machine for the agent lifecycle:
{
"states": ["PLAN", "ACT", "OBSERVE", "DONE", "ERROR"],
"transitions": {
"PLAN": ["ACT", "ERROR"],
"ACT": ["OBSERVE", "ERROR"],
"OBSERVE": ["PLAN", "DONE", "ERROR"]
},
"timeout_sec": {
"PLAN": 30,
"ACT": 120,
"OBSERVE": 30
}
}Each transition writes an event to the job log. If the agent stalls in ACT beyond timeout_sec, the supervisor forces ERROR and alerts. This gives you an audit trail and a way to build dashboards without instrumenting the model itself.
A supervisor loop enforces the machine:
async def supervisor(run_id, spec):
state = "PLAN"
while state not in ("DONE", "ERROR"):
started = time.monotonic()
state = await advance(state, run_id)
if time.monotonic() - started > spec["timeout_sec"][state]:
state = "ERROR"
await log_event(run_id, state)Backpressure and rate limits
Model endpoints are rate-limited. Even if you self-host, GPU memory caps concurrency. Unbounded async agent orchestration will happily spawn thousands of concurrent LLM calls and then drown in 429s.
Use a semaphore or a token bucket at the orchestration layer:
sem = asyncio.Semaphore(8)
async def bounded_llm_call(req):
async with sem:
return await gateway.chat(req)An inference gateway such as n4n.ai provides automatic fallback across providers when one is rate-limited or degraded, but your async agent orchestration still must throttle concurrent model calls to avoid queue blowup and cascading timeouts downstream. The gateway solves provider-level exhaustion; it does not solve your worker pool exhaustion.
For tool calls, apply the same discipline. A vector search service that allows 50 QPS will not survive 500 agents hitting it simultaneously. Put a shared limiter in front of every external dependency, not just the model.
Observability and replay
Distributed systems teach us to log events, not just state. Store each agent action as an append-only record:
# pseudo-schema for event store
echo '{"run_id":"r1","ts":1699000000,"state":"PLAN","model":"gpt-4o","tokens":120}' >> events.log
echo '{"run_id":"r1","ts":1699000030,"state":"ACT","tool":"sql","latency_ms":45}' >> events.logWith this log, you can replay a failed run in a sandbox, measure step latency distributions, and detect loops where OBSERVE returns to PLAN too many times. Without it, you are debugging by staring at prompts.
A minimal TypeScript client that emits these events alongside the model call keeps the concern out of the agent logic:
async function tracedChat(req: ChatRequest, runId: string) {
const res = await fetch("https://api.example.com/v1/chat/completions", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(req),
});
await appendEvent(runId, { state: "ACT", model: req.model, ts: Date.now() });
return res.json();
}Tradeoffs: complexity vs correctness
Adopting these patterns is not free. You now operate a database, a worker pool, and a supervisor. You write boilerplate for state transitions that a naive script avoided. On a team of one building a prototype, that overhead can be unjustified.
But the moment an agent touches real user data or runs longer than a human attention span, the naive approach costs more in incident response than the infrastructure ever did. The middle ground is to use a durable execution framework (Temporal, Restate, or a home-grown SQLite worker) from day one, so the pattern is default rather than retrofitted.
The other tradeoff is latency. Checkpointing after every step adds milliseconds of I/O. For a 90-second agent run, that is irrelevant. For a 200-millisecond chatbot, it is fatal—so do not use agent orchestration for synchronous UX; keep those on a tight request/response path and push long tasks to the async side.
Takeaway
Build async agent orchestration as if you are shipping a distributed service: durable job state, idempotent steps, explicit state machines, and enforced backpressure. Treat model calls and tool calls as unreliable network endpoints, because they are. The teams that survive scale are the ones who stopped trusting async to mean “safe” and started trusting persisted logs and recovery paths. If you internalize one lesson, make it this—an agent run is a workflow, not a function; design accordingly.