How to build a multi-agent system with LlamaIndex

Building a reliable llamaindex multi-agent system requires more than spawning parallel LLM calls; you need explicit tool boundaries, a supervisor that owns the plan, and a provider setup that degrades gracefully when a model is rate-limited. This walkthrough implements a researcher/writer/supervisor topology using LlamaIndex’s FunctionAgent and OpenAI-compatible LLMs, with code you can run locally.

Step 1: Install dependencies and configure the LLM

Start with a clean Python 3.11+ environment. Install the core LlamaIndex package and the OpenAI LLM adapter (LlamaIndex treats any OpenAI-compatible endpoint as an OpenAI instance).

pip install llama-index llama-index-llms-openai

For the model backend, point LlamaIndex at a single OpenAI-compatible endpoint. If you want automatic fallback across 240+ models when a provider is degraded, you can target n4n.ai’s gateway instead of a single vendor.

import os
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="gpt-4o-mini",
    api_key=os.environ["N4N_API_KEY"],
    api_base="https://api.n4n.ai/v1",  # OpenAI-compatible
    temperature=0.2,
)

The api_base swap is the only change required to get provider redundancy and per-token metering without touching agent logic.

Step 2: Define atomic tools for worker agents

Tools should be narrow, typed, and side-effect free where possible. LlamaIndex wraps Python functions with FunctionTool.from_defaults, which extracts the signature for the LLM.

from llama_index.core.tools import FunctionTool

def web_search(query: str) -> str:
    """Mock search returning static snippets."""
    return f"Top result for '{query}': LLMs benefit from retrieval grounding."

def word_count(text: str) -> int:
    """Count words in a string."""
    return len(text.split())

search_tool = FunctionTool.from_defaults(fn=web_search)
count_tool = FunctionTool.from_defaults(fn=word_count)

Keep the docstrings precise. The LLM routes on them, and vague descriptions cause agent confusion in a llamaindex multi-agent system.

Step 3: Build individual agents with narrow system prompts

FunctionAgent takes a name, system_prompt, tools, and llm. The name is how the supervisor references it later.

from llama_index.core.agent.workflow import FunctionAgent

def build_researcher(llm):
    return FunctionAgent(
        name="researcher",
        system_prompt="You gather facts using tools. Return concise bullet points.",
        tools=[search_tool],
        llm=llm,
    )

def build_writer(llm):
    return FunctionAgent(
        name="writer",
        system_prompt="You turn bullet points into a tight paragraph. No new facts.",
        tools=[count_tool],
        llm=llm,
    )

Each agent is single-purpose. A researcher that also writes defeats the isolation that makes a llamaindex multi-agent system debuggable.

Step 4: Implement supervisor orchestration via delegation

The simplest robust pattern is a supervisor agent whose tools are async functions that invoke worker agents. This avoids implicit handoff loops and gives you a clear call stack.

async def delegate_to_researcher(query: str) -> str:
    agent = build_researcher(llm)
    out = await agent.run(user_msg=f"Research: {query}")
    return str(out)

async def delegate_to_writer(bullets: str) -> str:
    agent = build_writer(llm)
    out = await agent.run(user_msg=f"Write from these: {bullets}")
    return str(out)

supervisor_tools = [
    FunctionTool.from_defaults(fn=delegate_to_researcher),
    FunctionTool.from_defaults(fn=delegate_to_writer),
]

supervisor = FunctionAgent(
    name="supervisor",
    system_prompt=(
        "Plan the task. Use delegate_to_researcher first, then delegate_to_writer. "
        "Return only the final paragraph."
    ),
    tools=supervisor_tools,
    llm=llm,
)

If you run LlamaIndex v0.11+, AgentWorkflow also supports native can_handoff_to handoffs, but explicit delegation is easier to unit test and log.

Step 5: Run the multi-agent workflow

Wrap execution in an async main. The supervisor will call the researcher tool, get bullets, then call the writer tool, then return the final text.

import asyncio

async def main():
    result = await supervisor.run(
        user_msg="Explain why retrieval grounding matters for agents."
    )
    print("FINAL:", str(result))

if __name__ == "__main__":
    asyncio.run(main())

A well-formed llamaindex multi-agent system will produce a paragraph that traces back to the researcher’s bullets, not hallucinated facts.

Step 6: Verify success end-to-end

Verification is not just “did it print something”. Add lightweight logging inside the tools to confirm the call graph.

def web_search(query: str) -> str:
    print(f"[tool] search called with: {query}")
    return f"Top result for '{query}': LLMs benefit from retrieval grounding."

Success criteria:

1. The supervisor invoked delegate_to_researcher exactly once.
2. The researcher called web_search.
3. The writer received the researcher’s output and returned a paragraph under 100 words.
4. No exception surfaced from the LLM provider.

Run the script and inspect stdout. If the supervisor tries to answer directly, tighten its system prompt and remove general knowledge tools from its scope.

Step 7: Production hardening

A demo topology is not a deployable llamaindex multi-agent system. Address these before shipping:

Timeouts and retries. Wrap agent.run in asyncio.wait_for with a 30s cap. LlamaIndex does not bound LLM latency by default.

Loop prevention. Pass max_iterations=8 to FunctionAgent to stop runaway tool calls.

Cost visibility. When using a gateway like n4n.ai, per-token usage metering is forwarded on each response; log response.raw or the usage field to track spend per agent.

Model routing. Honor client routing directives by setting extra_headers on the OpenAI client if you want the researcher on a cheap model and the writer on a stronger one. The gateway forwards provider cache-control hints, so mark static system prompts with cache_control to cut repeat token cost.

llm_researcher = OpenAI(
    model="mistralai/mixtral-8x7b-instruct",
    api_base="https://api.n4n.ai/v1",
    api_key=os.environ["N4N_API_KEY"],
    max_tokens=512,
)

Swap the LLM per agent as needed; the agent code does not change.

Where teams go wrong

The most common failure in a llamaindex multi-agent system is giving every agent the same broad toolset and hoping the supervisor “coordinates”. That produces redundant calls and contradictory state. Define tools per agent, keep the supervisor tool-only, and make handoffs explicit. The second failure is ignoring provider variability—when one model is flaky, the whole workflow stalls. An OpenAI-compatible endpoint with fallback removes that single point of failure without new code.

Follow the steps above, verify the call graph with logs, and you have a multi-agent pipeline that is observable, swappable, and ready for real traffic.