Visualizing multi-agent execution graphs for debugging

When a multi-agent pipeline returns garbage or hangs, reading interleaved logs wastes time. Visualizing multi-agent execution graphs turns opaque agent chatter into a navigable structure where you can see which planner called which tool, where retries happened, and where context drifted. This guide lays out a six-step path from raw agent logs to a graph you can reason about in minutes.

1. Instrument agents to emit span events

Start by treating every agent action as a trace span. A span has a start, end, parent, and metadata. Use OpenTelemetry if you already run a collector; otherwise, a JSONL file is enough to begin visualizing multi-agent execution graphs locally without standing up infrastructure.

Wrap each agent entrypoint with a context manager that writes a line before and after execution. Keep the instrumentation at the task level, not the function level.

import time, json, uuid, sys

def span(agent_name, parent_id=None):
    sid = uuid.uuid4().hex[:8]
    start = time.time()
    print(json.dumps({"type": "span_start", "id": sid,
                       "parent": parent_id, "agent": agent_name,
                       "start": start}), flush=True)
    class Ctx:
        def __enter__(self): return self
        def __exit__(self, *a):
            print(json.dumps({"type": "span_end", "id": sid,
                              "dur": time.time() - start}), flush=True)
    return Ctx()

Call it at the top of each agent’s run method. Pass the parent span id when one agent invokes another. In async code, store the current span id in a contextvars.ContextVar so child tasks inherit it correctly. The pitfall here is nesting too deeply: a span per token or per retry is noise. Span at agent boundaries only.

2. Model the graph as nodes and edges

After a run, you have a stream of events. Convert them to a directed graph. Each span_start becomes a node; each parent link becomes a directed edge. Include status and token counts if your LLM client exposes them, because those fields drive later cost views.

A minimal node schema:

{
  "id": "a1b2c3",
  "parent": "000000",
  "agent": "planner",
  "start": 1710000000.1,
  "dur": 1.24,
  "tokens": {"prompt": 1200, "completion": 300},
  "status": "ok"
}

Build the graph in memory with a dict keyed by id. If you see a node with no parent and it isn’t the root, you have an orphan—usually a background task that escaped instrumentation. Edge attributes matter too: label an edge with the call type (tool_call, llm_completion, handoff) so the renderer can style them differently.

3. Render with Mermaid for fast iteration

For a debugging session, you don’t need a custom React app. Mermaid renders directed graphs from text and sits inside any Markdown viewer or CI artifact. Generate a graph TD from your nodes:

def to_mermaid(nodes):
    lines = ["graph TD"]
    for n in nodes:
        label = f"{n['agent']}<br>{n.get('status','?')}"
        lines.append(f"  {n['id']}[{label}]")
        if n.get("parent"):
            lines.append(f"  {n['parent']} --> {n['id']}")
    return "\n".join(lines)

Paste the output into a .md file or a Mermaid live editor. Use classDef to color nodes by status—red for error, yellow for timeout. The tradeoff: Mermaid degrades past ~200 nodes. For larger graphs, use Graphviz dot for static output or Cytoscape.js when you need pan and zoom. Pick the renderer based on graph size, not preference.

4. Attribute LLM cost and routing to nodes

Multi-agent systems live and die on model calls. If each agent talks to a different provider, correlating spend with graph nodes is messy. An OpenAI-compatible gateway such as n4n.ai returns per-token usage and honors client routing directives, so you can attach usage to a span without writing per-vendor code. It also forwards provider cache-control hints, letting you mark cache hits directly on the node.

Parse the response object uniformly regardless of backend:

# OpenAI-style response from any compliant gateway
usage = resp.get("usage", {})
node["tokens"] = {
    "prompt": usage.get("prompt_tokens", 0),
    "completion": usage.get("completion_tokens", 0)
}
node["cache_hit"] = usage.get("prompt_tokens_details", {}).get("cached_tokens", 0) > 0

Now your visualization can show which planner subtree burned 80% of the budget, or which retriever silently paid for cache misses. If a provider is rate-limited, an automatic fallback produces a second edge from the same node; render that as a dashed line so retries are visible.

5. Detect anomalies from topology

With the graph in hand, look for structural bugs before reading any payload:

Cycles: agent A calls B, B calls A. Usually a missing termination condition.
Fan-out without aggregation: many workers, no reducer node collecting results.
Bottleneck: one node with 50 children and 30s duration while siblings idle.

Write a checker that runs on every trace:

from collections import defaultdict
def find_cycles(edges):
    adj = defaultdict(list)
    for p, c in edges: adj[p].append(c)
    visited = set(); stack = set()
    def dfs(u):
        if u in stack: return True
        if u in visited: return False
        stack.add(u)
        for v in adj[u]:
            if dfs(v): return True
        stack.remove(u); visited.add(u)
        return False
    return any(dfs(n) for n in adj)

Run this in CI on recorded traces. It catches recursion bugs faster than any log grep. For timeout analysis, sort nodes by dur and inspect the top five—usually the fix is prompt compression or a smaller model, not a code change.

6. Build a replayable debug loop

Store traces as JSONL in object storage or a local log dir. When a bug report arrives, pull the trace, filter to the suspect agent, and re-render. Filtering avoids the 200-node Mermaid limit and keeps the view focused.

grep '"agent":"retriever"' run_123.jsonl > sub_trace.jsonl

Then load and convert:

import json
nodes = [json.loads(l) for l in open("sub_trace.jsonl") if '"span_start"' in l]
print(to_mermaid(nodes))

Keep the rendering script in your repo as viz_graph.py. Engineers should run python viz_graph.py run_123.jsonl > graph.md in under five seconds. Add a --status error flag to filter only failed nodes. The goal is a loop: instrument, capture, render, inspect, fix, repeat.

Common pitfalls

Over-instrumenting. Spanning every internal function creates a hairball. Span at agent boundaries only.

Ignoring parallel branches. In async systems, spans overlap. Use wall-clock start/end and draw edges by parent id, not by log order. A linear log reader will mislead you about concurrency.

Missing token data. If you don’t capture usage at call time, you can’t attribute cost later. Instrument the LLM client wrapper once, not per agent.

Static only. A PNG is fine for a postmortem, but interactive zoom saves time during active debugging. Switch to Cytoscape when the graph exceeds Mermaid’s comfort zone.

No retention policy. Traces grow fast. Keep last 100 runs per agent version; older ones rarely help and just cost storage.

Visualizing multi-agent execution graphs is not a one-time setup. It is a debugging discipline that pays back the first time a planner loops or a worker silently fails. Ship the instrumentation with the agents, and the graph will be there when the pager goes off.