Debugging a system where several LLM agents hand off tasks to each other is painful without visibility. To trace multi-agent workflows effectively, you need distributed tracing that ties each agent’s reasoning, tool calls, and model invocations into a single causal graph.
Step 1: Establish a single trace per workflow invocation
Start a root span the moment a workflow enters your system. Every downstream agent must inherit this context. In Python, OpenTelemetry’s Context and Tracer make this straightforward. Initialize the tracer provider once at process startup, then create a root span in your entrypoint (API route, queue consumer, or CLI).
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
tracer = trace.get_tracer("workflow.orchestrator")
def handle_task(task_id: str, payload: dict):
with tracer.start_as_current_span("workflow.root") as span:
span.set_attribute("workflow.id", task_id)
span.set_attribute("workflow.type", "multi_agent")
# contextvars carry this span to any sync/async callee
run_agents(payload)If agents run in separate processes or services, serialize the context into W3C traceparent headers. The opentelemetry-propagate module handles this. Without a single root, you will not trace multi-agent workflows as one tree—you’ll get disconnected fragments that force you to manually correlate logs by timestamp.
Set a deterministic sampler in staging so every workflow is recorded. In production, use a parent-based sampler that keeps traces where the root was sampled, and add an error-only tail sampler if cost is a concern.
Step 2: Wrap each agent’s reasoning loop in a child span
Each agent should open a child span as soon as it receives a task. Use the current context (propagated from Step 1) so the span nests correctly. Record the agent identity and model up front; you’ll thank yourself when filtering in the UI.
def run_agent(agent_name: str, model: str, instruction: str):
with tracer.start_as_current_span(f"agent.{agent_name}") as span:
span.set_attribute("agent.name", agent_name)
span.set_attribute("agent.model", model)
result = agent_loop(instruction, model)
span.set_attribute("agent.result_len", len(result))
return resultFor async agents, OpenTelemetry’s contextvars integration propagates across asyncio tasks automatically. Just avoid manually creating Context objects that shadow the current one.
async def run_agent_async(agent_name, model, instruction):
with tracer.start_as_current_span(f"agent.{agent_name}") as span:
span.set_attribute("agent.model", model)
return await agent_loop_async(instruction, model)Keep the span open for the entire agent turn, not just the LLM call. Sub-spans for individual LLM requests and tool invocations will nest underneath automatically, giving you a clear flame graph of each agent’s internal steps.
Step 3: Emit spans for LLM calls and tool executions
The most useful telemetry comes from leaf operations. For an LLM call, create a span that captures token counts, latency, and the model used. If you call an OpenAI-compatible endpoint directly, wrap the client call:
def call_llm(messages, model):
with tracer.start_as_current_span("llm.call") as span:
span.set_attribute("llm.model", model)
span.set_attribute("llm.input_messages", len(messages))
resp = openai_client.chat.completions.create(model=model, messages=messages)
span.set_attribute("llm.prompt_tokens", resp.usage.prompt_tokens)
span.set_attribute("llm.completion_tokens", resp.usage.completion_tokens)
return resp.choices[0].message.contentFor streaming responses, emit span events as chunks arrive and set final token counts when the stream closes. Tool calls deserve their own spans too. Include inputs and outputs as attributes (truncated if large):
def use_tool(tool_name, args):
with tracer.start_as_current_span(f"tool.{tool_name}") as span:
span.set_attribute("tool.args", json.dumps(args)[:500])
output = dispatch_tool(tool_name, args)
span.set_attribute("tool.output", output[:500])
return outputWhen one agent delegates to another, treat the handoff as a span that carries the sub-agent’s name and the transmitted context. This is the core of how you trace multi-agent workflows across boundaries.
Step 4: Propagate context across network boundaries
If your agents communicate over HTTP (common in micro-agent architectures), inject the trace context into request headers. Using httpx and OTel propagators:
from opentelemetry.propagate import inject
import httpx
def call_remote_agent(url, task):
headers = {}
inject(headers) # adds traceparent, tracestate
resp = httpx.post(url, json=task, headers=headers)
return resp.json()The receiving service must extract the context before starting its agent span:
from opentelemetry.propagate import extract
def receive_task(request):
ctx = extract(request.headers)
with tracer.start_as_current_span("agent.remote", context=ctx) as span:
# ...For message queues, store the propagated headers in message metadata. Skip this and your trace multi-agent workflows effort collapses into per-service silos where you can’t tell which upstream call triggered a slow sub-agent.
Step 5: Aggregate traces and join with provider metadata
Run an OpenTelemetry collector to receive spans and forward them to a backend like Jaeger or Tempo. A minimal collector config:
receivers:
otlp:
protocols:
grpc:
http:
exporters:
jaeger:
endpoint: "jaeger:14250"
service:
pipelines:
traces:
receivers: [otlp]
exporters: [jaeger]If you route model calls through a gateway such as n4n.ai, which honors client routing directives and forwards provider cache-control hints, attach the same trace ID to the X-Trace-Id header. Its per-token usage metering can then be joined with your spans by timestamp and trace ID, giving you cost attribution per agent step without custom instrumentation.
Bring up a local stack for verification:
docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-onePoint your OTEL_EXPORTER_OTLP_ENDPOINT at http://localhost:4317 and spans will appear in the Jaeger UI at localhost:16686.
Step 6: Verify success with a controlled test
Build a two-agent workflow: Agent A answers a question, then delegates to Agent B for a calculation. Run it once and open your trace backend.
def run_agents(payload):
a_out = run_agent("researcher", "gpt-4o-mini", payload["query"])
b_out = run_agent("calculator", "gpt-4o-mini", a_out)
return b_outVerification checklist:
- A single root span
workflow.rootexists. - Under it,
agent.researcherandagent.calculatorspans appear sequentially. - Each agent span contains nested
llm.callandtool.*spans with token attributes. - The
traceparentheader is present in any inter-agent HTTP logs. - Token counts in spans match the provider’s billed usage (or gateway metering).
If any span is missing, check that the tracer provider is initialized before agent code runs and that context propagation isn’t being overwritten by a new Context.
Step 7: Analyze traces to locate handoff failures
Once you can reliably trace multi-agent workflows, use the backend to find where things break. In Jaeger, search by workflow.id and look at the span timeline. A common failure mode is a delegated agent span that starts but never emits an llm.call child—indicating the agent crashed before model invocation. Another is a sudden spike in llm.completion_tokens on a single step, revealing a prompt loop bug.
Add a saved view for error=true spans and alert on workflows where root span duration exceeds a threshold but no leaf span is marked erroneous. That usually means a silent timeout in an un-instrumented network call.
Common pitfalls
Sampling at 1% in production hides the exact path of a failing handoff. Use head-based sampling for errors or trace every workflow in staging.
Another mistake: logging instead of tracing. Logs are linear and lose causal structure. Emit spans, not just print statements.
Finally, avoid putting raw user PII in span attributes. Truncate or hash; OpenTelemetry supports redaction processors.
Reusable instrumentation pattern
A decorator keeps instrumentation consistent across agents:
def traced_agent(name, model):
def decorator(fn):
def wrapper(*args, **kwargs):
with tracer.start_as_current_span(f"agent.{name}") as span:
span.set_attribute("agent.name", name)
span.set_attribute("agent.model", model)
return fn(*args, **kwargs)
return wrapper
return decorator
@traced_agent("researcher", "gpt-4o-mini")
def researcher_agent(query):
# ...Follow these steps and you will trace multi-agent workflows with enough fidelity to debug handoffs, spot runaway token spend, and reconstruct exactly which agent produced a bad answer.