Most LLM agent systems start as a single HTTP request that calls a model, runs a tool, and returns an answer. The choice between a sync vs async agent architecture determines whether that request blocks until completion or yields a handle to poll later—a fork that affects timeout handling, cost attribution, and failure recovery. This article compares both approaches across the dimensions that matter when you ship to production.
The core difference
A synchronous agent architecture executes the entire reasoning loop inside one blocking call. The client sends a prompt, the server invokes the model, calls tools, possibly loops, and returns the final text. The connection stays open (or the RPC waits) for the full duration.
An asynchronous agent architecture acknowledges the request immediately and processes the work out of band. The client receives a task ID or webhook URL, then fetches results later. The agent runtime persists state between model calls.
# Sync: blocks for the full agent run
def sync_agent(query: str) -> str:
msg = [{"role": "user", "content": query}]
for _ in range(5): # max reasoning steps
resp = client.chat.completions.create(model="gpt-4o", messages=msg, tools=TOOLS)
if resp.choices[0].finish_reason == "stop":
return resp.choices[0].message.content
msg.append(resp.choices[0].message)
msg.append(run_tool(resp.choices[0].message.tool_calls[0]))
return "max steps exceeded"# Async: submit and poll
job = requests.post("/agents/run", json={"query": query}).json()
job_id = job["id"]
while True:
status = requests.get(f"/agents/status/{job_id}").json()
if status["state"] == "done":
return status["result"]
time.sleep(2)The sync vs async agent architecture split is not about model capability; it is about control flow and where the waiting happens.
Capabilities
Sync agents handle linear, low-step tasks well: a retrieval-augmented answer, a single tool call, a quick classification. They cannot survive a process restart mid-run, and they fall over if a tool takes minutes.
Async agents support long horizons: multi-hour research, human approval gates, parallel subagent fan-out. Because state lives in a store, you can resume after a crash, add a review step, or spawn child tasks that report back to a parent.
# Async subagent spawn
parent_task = task_queue.create(workflow="research", input=query)
task_queue.spawn(child="fetch_prices", parent=parent_task.id)
task_queue.spawn(child="summarize_docs", parent=parent_task.id)If your agent must call a rate-limited API that returns in 30 seconds, sync will tie up a worker thread; async lets you yield and recharge.
Cost model
Token cost is identical for the same conversation regardless of sync vs async agent architecture—the model sees the same messages. The difference is operational overhead.
Sync runs incur no extra storage or queuing cost. You pay only for the inference tokens and the wall-clock compute of the serving process.
Async runs need a state store (Redis, Postgres), a queue, and often duplicated context reloads when a worker picks up a paused task. Those infrastructure costs are small per job but real at scale. When you route through a gateway like n4n.ai, per-token usage metering applies uniformly, but async systems must tag each call with the job ID to attribute spend correctly.
client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=msg,
metadata={"job_id": job_id} # custom header for billing correlation
)Budget for async: queue hosting, state serialization, and occasional replay of truncated histories.
Latency and throughput
Sync latency equals the full agent runtime. A 20-step agent with 2s per model call shows 40s+ to the user. Throughput is limited by concurrent blocking workers; a Flask sync server with 10 threads handles 10 concurrent agents.
Async decouples perceived latency from actual work. The HTTP POST returns in 50ms with a task ID. Actual completion may be slower due to queue delays, but you can scale workers independently of inbound connections. A single gateway can accept thousands of submissions while 100 workers drain the queue.
For batch workloads—say, 10k docs to summarize—async wins decisively. Sync wins for interactive chat where the user expects a streaming token flow and abort-on-disconnect.
Ergonomics and developer experience
Sync code reads top-to-bottom. Debugging is a standard stack trace; you can step through the loop in a notebook. Testing is a single function call.
Async demands idempotent steps, explicit state schemas, and recovery logic. A developer must handle “what if the worker dies after the tool call but before saving?” Temporal or LangGraph help, but the mental model is a distributed system.
# Async step must be idempotent
def step_fetch(task_id):
if cache.exists(f"fetched:{task_id}"):
return cache.get(f"fetched:{task_id}")
data = slow_api()
cache.set(f"fetched:{task_id}", data)
return dataSync vs async agent architecture also changes observability: sync logs are per-request; async needs distributed tracing across task hops.
Ecosystem and tooling
Sync agents fit any web framework: FastAPI, Express, Rails. Most SDKs assume request-response.
Async agents lean on workflow engines: Temporal, Celery, Inngest, LangGraph, or custom Kubernetes operators. Model providers now emit partial streaming and cancel tokens that map cleanly to async cancellation. The tooling is heavier but built for exactly this.
If you already run a job queue for non-AI work, async agents are a small addition. If you are a solo dev shipping a Slack bot, sync keeps the stack small.
Limits and failure modes
Sync limits: reverse proxy timeouts (30s on many PaaS), memory growth if context balloons, no resume after crash. A stuck tool call hangs the worker.
Async limits: orphaned tasks, poisoned queues, state store outages that lose in-flight agents. A bug in resume logic can double-charge tools or loop forever. You must implement TTLs and dead-letter queues.
# Async safety net
task_queue.configure(ttl=3600, dead_letter="failed_agents")Sync fails fast and locally; async fails subtly and at 3am.
Side-by-side comparison
| Dimension | Sync agent architecture | Async agent architecture |
|---|---|---|
| Capabilities | Linear, short-horizon, in-process tools | Long-running, human-in-loop, parallel subagents |
| Cost model | Only tokens + worker CPU | Tokens + queue + state store + replay overhead |
| Latency / throughput | Blocks on full run; limited by worker threads | Immediate ack; scales via independent workers |
| Ergonomics | Straight-line code, easy debug | State machines, idempotency, distributed tracing |
| Ecosystem | Any HTTP framework, standard SDKs | Temporal, Celery, LangGraph, queue infra |
| Limits | Proxy timeouts, no crash recovery | Orphan tasks, state loss, resume bugs |
Which to choose
Choose sync if:
- Your agent completes in under 20–30 seconds.
- You need streaming tokens to a live user.
- The team is small and the workflow is a single linear pass.
- Infrastructure simplicity outweighs horizontal scale.
Example: a support bot that retrieves one doc and answers, or a code review labeler.
Choose async if:
- Agents run minutes to hours (deep research, batch enrichment).
- You require human approval or external webhooks mid-run.
- Throughput matters more than per-request simplicity.
- You already operate a queue and can absorb state store cost.
Example: a nightly pipeline that spins 500 subagents to analyze competitors, or an agent that waits for a user email reply before finalizing a contract.
The sync vs async agent architecture decision is not permanent. Many production systems start sync, then extract the long-tailed steps into an async worker while keeping the interactive core synchronous. Design the agent loop so the orchestration logic is independent of the transport, and you can shift later without rewriting the reasoning.