How multi-agent systems handle disagreement between agents

Multi-agent systems rarely agree by default. Effective multi-agent disagreement resolution determines whether your orchestration layer produces coherent output or collapses into contradictory noise. The thesis here is simple: treat disagreement as a first-class signal, select a resolution protocol based on task criticality, and encode explicit convergence criteria before you ship.

Why disagreement is inevitable

You spin up three agents to draft a policy document. One cites a regulation that expired; another ignores it; the third hallucinates a clause. None are “wrong” from their local prompt perspective. LLM agents are stochastic, context-bound, and optimized for plausibility, not consensus.

In a single-agent system, contradiction is internal and often invisible. In multi-agent orchestration, contradictions surface as explicit divergent outputs. That visibility is a feature. It forces you to handle uncertainty instead of burying it.

The cost of ignoring disagreement is silent failure. A naive “take the first response” pipeline will occasionally ship garbage when agent one is the unreliable one. Multi-agent disagreement resolution is the engineering discipline of making that uncertainty actionable.

Core patterns for multi-agent disagreement resolution

Four patterns cover most production needs. Each trades latency and cost for robustness differently.

Majority voting and its limits

The simplest protocol: run N agents, parse their answers into discrete choices, take the mode. For classification or extractive tasks (e.g., “is this invoice valid?”), this works well.

from collections import Counter

def resolve_vote(responses: list[str]) -> str:
    # assume each response is a normalized label
    counts = Counter(responses)
    return counts.most_common(1)[0][0]

Voting fails when answers are not discretely comparable. Free-form text rarely votes cleanly. If two agents say “refund approved with 10% penalty” and one says “refund denied”, the mode is meaningless. Use voting only when you can deterministically bucket outputs.

Structured debate with a moderator

For open-ended tasks, a moderated debate converges better than voting. Agents exchange critiques for K rounds; a moderator agent synthesizes. The moderator can be a smaller, cheaper model tasked only with detecting consensus.

Key design choice: limit rounds. Unbounded debate burns tokens and rarely improves after round two.

Delegation to a referee agent

When agents disagree on a factual claim, escalate to a referee with stricter prompting or tool access (e.g., web search). The referee’s verdict is final. This is appropriate when one agent has tools the others lack.

async def referee(claim_a: str, claim_b: str) -> str:
    prompt = f"Verify which claim is correct using tools. A: {claim_a} B: {claim_b}"
    return await call_llm(system="You are a precise verifier.", user=prompt, tools=["web_search"])

Programmatic reconciliation

Sometimes the best resolver is not an LLM but code. If agents output JSON with confidence scores, merge by score. If they output coordinates, average them. This is the most deterministic and cheapest option, but requires schema discipline.

{
  "agent_1": {"value": 42, "confidence": 0.7},
  "agent_2": {"value": 38, "confidence": 0.9}
}

Weighted merge: (42*0.7 + 38*0.9) / (0.7+0.9) = 39.75. No model call needed.

Implementing a debate protocol

Below is a minimal asyncio orchestrator that runs agents, detects disagreement via embedding distance, and triggers one debate round. It assumes call_llm and embed are available.

import asyncio

async def run_agent(prompt: str, model: str) -> str:
    return await call_llm(model=model, user=prompt)

async def detect_disagreement(responses: list[str]) -> bool:
    vecs = [await embed(r) for r in responses]
    # crude pairwise check
    for i in range(len(vecs)):
        for j in range(i+1, len(vecs)):
            if cosine(vecs[i], vecs[j]) < 0.8:
                return True
    return False

async def debate_round(responses: list[str], model: str) -> list[str]:
    tasks = []
    for r in responses:
        crit = f"Critique this peer output concisely: {r}"
        tasks.append(run_agent(crit, model))
    return await asyncio.gather(*tasks)

async def orchestrate(prompts: list[str], models: list[str]):
    initial = await asyncio.gather(*[run_agent(p, m) for p, m in zip(prompts, models)])
    if not await detect_disagreement(initial):
        return initial[0]
    debated = await debate_round(initial, models[0])
    if await detect_disagreement(debated):
        # final fallback: referee
        return await referee(debated[0], debated[1])
    return debated[0]

If you run agents on different model families for capability or cost reasons, route through a gateway that honors client routing directives and forwards provider cache-control hints. n4n.ai exposes a single OpenAI-compatible endpoint covering 240+ models with automatic fallback, which keeps the above call_llm unchanged when a provider is rate-limited.

Tradeoffs: latency, cost, and determinism

Every multi-agent disagreement resolution step multiplies token spend. A debate with 3 agents and 2 rounds costs ~6 generation calls plus embedding. Voting costs N calls but no extra. Referee adds one more call but can use a cheaper model.

Latency tracks calls linearly unless you parallelize. The asyncio example parallelizes initial and critique rounds, but the referee is sequential.

Determinism is the hidden tax. Voting on discrete labels is reproducible; debate output varies per run. If your system requires auditability, log the full transcript and the resolution path.

When to use what

Decisive takeaway

Build disagreement handling into the orchestration contract from day one. Pick the cheapest resolver that matches your output schema, cap debate rounds at two, and always log the divergence. Multi-agent disagreement resolution is not a research problem; it is a plumbing problem with known fittings. Ship the plumbing before you scale the agents.