A multi-agent system is a collection of autonomous AI agents that perceive, reason, and act in a shared environment to achieve individual or collective goals. Each agent maintains its own state, policy, and communication interface, coordinating through explicit message passing, shared memory, or environmental signals rather than a central controller. Understanding how multi-agent AI systems work is essential for engineers moving beyond single-model pipelines into architectures where specialization, parallelism, and fault isolation matter.
How multi-agent systems work
At the architectural level, a multi-agent system decomposes a complex task into roles — planner, researcher, coder, critic, executor — each instantiated as a separate agent with a focused prompt, toolset, and context window. Agents communicate via structured protocols: JSON-over-HTTP, gRPC, or message buses like NATS or Kafka. The orchestration layer (often called a supervisor or conductor) handles task decomposition, routing, aggregation, and failure recovery.
A minimal agent loop looks like this:
class Agent:
def __init__(self, name: str, system_prompt: str, tools: list[Tool]):
self.name = name
self.system_prompt = system_prompt
self.tools = {t.name: t for t in tools}
self.history: list[Message] = []
async def step(self, input: str) -> str:
self.history.append(Message(role="user", content=input))
response = await self.llm.chat(
messages=[Message(role="system", content=self.system_prompt)] + self.history,
tools=[t.schema for t in self.tools.values()],
)
if response.tool_calls:
for call in response.tool_calls:
result = await self.tools[call.name].execute(call.arguments)
self.history.append(Message(role="tool", content=result, tool_call_id=call.id))
return await self.step("") # recurse with tool results
self.history.append(Message(role="assistant", content=response.content))
return response.contentAgents share context through three primary patterns:
Explicit message passing — Agents serialize state into structured messages (JSON, Protocol Buffers) and send them over a transport. This is the most debuggable pattern; you can inspect, replay, and version every interaction.
Shared scratchpad — A mutable key-value store (Redis, etcd, even a Postgres table) where agents write intermediate artifacts: search results, code diffs, extracted entities. Requires careful concurrency control — optimistic locking or CRDTs — to avoid lost updates.
Environmental signaling — Agents observe side effects of each other’s actions in a shared environment (a filesystem, a Kubernetes cluster, a browser DOM). This is implicit coordination; it scales well but makes causality harder to trace.
The supervisor pattern is the most common orchestration topology:
class Supervisor:
def __init__(self, agents: dict[str, Agent], router: Router):
self.agents = agents
self.router = router # decides which agent acts next
async def run(self, task: str) -> Result:
context = {"task": task, "artifacts": {}, "history": []}
while not self.is_complete(context):
agent_name = await self.router.select(context)
agent = self.agents[agent_name]
output = await agent.step(self.format_input(context, agent_name))
context = self.update_context(context, agent_name, output)
return self.aggregate(context)The router can be an LLM (planning), a rule engine (deterministic), or a hybrid. Hybrid routers — LLM for high-level decomposition, rules for low-level dispatch — tend to be the most reliable in production.
Why multi-agent systems matter
Single-agent systems hit hard limits on context window, tool diversity, and error recovery. A 128k context window sounds large until you feed it a codebase, API docs, test output, and a multi-step reasoning trace. Multi-agent architectures solve this through context isolation: each agent sees only what it needs, reducing token spend and hallucination surface area.
They also enable specialized tooling. A researcher agent needs web search and PDF parsing; a coder agent needs a language server, git, and a test runner; a critic agent needs static analysis and security scanners. Bundling all tools into one agent bloats the prompt and confuses the model. Separation lets you tune each agent’s temperature, model tier, and prompt independently.
Parallelism is the third lever. Independent sub-tasks — searching multiple APIs, generating test cases for different modules, reviewing separate PRs — can run concurrently across agents. The supervisor aggregates results, cutting wall-clock time roughly by the fan-out factor (minus coordination overhead).
Fault isolation matters more than it sounds. When a coder agent hallucinates a library function, the critic catches it before the executor runs the code. When a researcher hits a rate limit, the supervisor can retry with a different provider or degrade to cached results. In a monolithic agent, one failure cascades into the entire context.
Concrete example: automated code review pipeline
Consider a pipeline that reviews pull requests for correctness, style, and security. Four agents, one supervisor:
agents:
- name: context-gatherer
model: gpt-4o-mini
tools: [git_diff, fetch_linked_issues, read_changed_files]
prompt: |
Extract the minimal context needed to review this PR.
Output: {files: [], issues: [], dependencies: []}
- name: static-analyzer
model: gpt-4o
tools: [run_linter, run_typecheck, run_security_scan]
prompt: |
Run all static analysis tools on the changed files.
Categorize findings: blocking, warning, nitpick.
Output: {findings: []}
- name: logic-reviewer
model: o1
tools: [read_file, search_codebase, run_tests]
prompt: |
Review the logic for correctness, edge cases, and test coverage.
Focus on: business logic, concurrency, data integrity.
Output: {comments: [], suggested_changes: []}
- name: summarizer
model: gpt-4o-mini
tools: [post_pr_comment]
prompt: |
Synthesize all agent outputs into a single review comment.
Group by file, prioritize blocking issues.
Output: {comment_markdown: string}The supervisor routes: context-gatherer → (static-analyzer, logic-reviewer in parallel) → summarizer. Each agent writes to a shared Redis hash keyed by PR number. The summarizer reads the aggregate, posts the comment, and marks the run complete.
This pipeline runs in ~90 seconds for a typical PR. A single-agent equivalent with all tools loaded takes 3-4 minutes and produces noisier reviews because the model must constantly context-switch between linting, logic analysis, and formatting.
Common misconceptions
Misconception: “Multi-agent just means multiple prompts.” A prompt chain is not a multi-agent system. The defining characteristic is autonomy — each agent decides its next action based on its own policy and observation, not a fixed script. If your “agents” are just functions called in sequence by a deterministic orchestrator, you have a pipeline, not a multi-agent system. Pipelines are fine; call them what they are.
Misconception: “More agents = better results.” Every additional agent adds latency, failure modes, and coordination complexity. The sweet spot for most production workloads is 3-6 agents. Beyond that, you spend more time debugging message formats and deadlocks than solving the original problem. Start with one agent, split only when you hit a concrete limit: context overflow, tool conflict, or latency budget.
Misconception: “Agents should use natural language to talk to each other.” Natural language is lossy, non-deterministic, and expensive to parse. Define typed schemas (Pydantic, Protocol Buffers, JSON Schema) for inter-agent messages. Validate on send and receive. Reserve natural language for human-facing outputs. This single discipline eliminates a class of “the agent misunderstood the other agent’s output” bugs.
Misconception: “The supervisor should be the smartest model.” The supervisor makes high-frequency, low-complexity decisions: routing, aggregation, termination. It benefits from low latency and high reliability, not raw reasoning power. Use a fast, cheap model (GPT-4o-mini, Claude 3.5 Haiku, Llama 3.1 8B) for the supervisor. Reserve your reasoning budget for the specialist agents doing the actual work.
Misconception: “Multi-agent systems require a framework.” LangGraph, AutoGen, CrewAI, and others accelerate prototyping. In production, they often become liability — opaque control flow, hidden retries, framework-specific state serialization. Many teams graduate to a thin custom supervisor (200-400 lines) plus standard message infrastructure. The framework’s value is in the patterns it teaches, not the code it generates.
Operational realities
Observability is non-negotiable. You need distributed tracing (OpenTelemetry) across agent boundaries. Every message should carry a trace ID. Log the full input/output for each agent step — this is your debugging dataset and your eval corpus.
Evaluation shifts from “does the model answer correctly?” to “does the system achieve the goal?” Build end-to-end evals with golden trajectories. Measure: task success rate, token cost per task, latency p50/p99, agent-specific error rates. Regression test when you change prompts, models, or routing logic.
Cost control requires per-agent budgets. A logic-reviewer on o1 can spend $2-5 per PR. Put hard limits: max tokens per step, max steps per run, max wall-clock time. The supervisor should enforce these and emit metrics when agents hit ceilings.
Provider diversity matters at scale. If your coder agent depends on a single model provider and that provider degrades, your pipeline stalls. Routing layers that support automatic fallback across providers — honoring cache-control hints and per-token metering — keep the system moving. This is where a gateway like n4n.ai earns its keep: one OpenAI-compatible endpoint addressing 240+ models with built-in fallback and usage accounting.
When to use multi-agent (and when not to)
Use multi-agent when:
- The task decomposes cleanly into distinct roles with different tool requirements
- Context isolation improves quality or reduces cost
- Parallel execution meaningfully reduces latency
- You need fault isolation between failure-prone steps
Stick with single-agent (or prompt chains) when:
- The task is linear and context fits in one window
- You need deterministic, auditable execution (regulatory, safety-critical)
- Team lacks operational maturity for distributed systems debugging
- Latency budget is under 5 seconds end-to-end
The best architecture is the simplest one that meets your constraints. Multi-agent is a power tool — reach for it when the work demands it, not because it’s trendy.