Why multi-agent tracing is harder than single-call tracing

Single-call LLM tracing is a solved problem: you capture the request, the response, and the latency. Multi-agent tracing challenges arise the moment you orchestrate multiple models, tools, and asynchronous handoffs, because the system’s behavior is no longer a linear request-response pair but a dynamic graph with branching, merging, and retries.

The baseline: single-call tracing

A standard completion call produces one logical span. You record the model, the token counts, and the round-trip time. Most APM tools ingest this without custom code.

{
  "trace_id": "a1b2c3",
  "spans": [
    {
      "span_id": "s1",
      "name": "chat.completion",
      "start": "2024-05-01T10:00:00Z",
      "end": "2024-05-01T10:00:02Z",
      "attributes": {
        "model": "gpt-4o",
        "prompt_tokens": 120,
        "completion_tokens": 30
      }
    }
  ]
}

That representation is sufficient because there is exactly one entry point and one exit point. The trace is a line.

What breaks: concurrency and async handoffs

Multi-agent systems introduce parallel execution. A coordinator agent dispatches three specialized workers, then aggregates their outputs. The spans overlap in time and share a parent, but each worker may itself call a model, a retrieval tool, or another agent. The moment you use asyncio.gather or a task queue, the strict nesting of synchronous spans collapses.

import asyncio
from opentelemetry import trace

tracer = trace.get_tracer("multiagent")

async def coordinator():
    with tracer.start_as_current_span("coordinator") as span:
        results = await asyncio.gather(
            worker("researcher"),
            worker("critic"),
            worker("synthesizer")
        )
        return aggregate(results)

async def worker(role):
    with tracer.start_as_current_span(f"agent.{role}") as span:
        # each worker calls an LLM and tools
        return await run_subtask(role)

The trace is now a tree (or DAG) where timing is non-contiguous. Single-call tracers that assume synchronous, nested calls will either flatten the concurrency or drop the parent-child links. You need context propagation that survives asyncio.gather and crosses event loops. If you spawn agents dynamically based on intermediate results, the span tree is not even known at request start—it grows as the system reasons.

Example: a coordinator and three workers

Consider a research pipeline: the researcher fetches documents, the critic evaluates relevance, the synthesizer writes the answer. The critic may start before the researcher finishes if streaming is used. A flame graph shows gaps and overlaps. If you only log per-call latency, you miss that the critic was blocked waiting on a shared scratchpad lock—a contention issue invisible in single-call traces. Worse, if the researcher retries its retrieval three times, those retries appear as siblings, not children, unless your instrumentation explicitly links them.

Non-deterministic routing and model abstraction

In a single call, the model field is static. In multi-agent flows, a router selects the model at runtime based on cost, latency, or payload size. The requested alias ("fast") may resolve to different physical models across runs. A prompt that worked yesterday may silently route to a different provider today.

When you route through a single OpenAI-compatible endpoint that fronts 240+ models with automatic fallback, the span must record the resolved provider and the actual model served, not just the requested alias. Otherwise, debugging a quality regression becomes impossible.

span.set_attribute("requested_model", "router-default")
span.set_attribute("resolved_model", "anthropic/claude-3.5-sonnet")
span.set_attribute("provider_fallback", True)
span.set_attribute("cache_hit", False)

Without those attributes, you cannot distinguish a prompt engineering bug from a silent fallback to a weaker model during a provider outage. You also lose the ability to honor client routing directives in post-hoc analysis—did the caller request a specific vendor and get ignored?

Context propagation and state mutations

Agents communicate through shared state: a message list, a vector store, a scratchpad. A trace that only captures LLM calls misses why an agent made a decision. You must emit span events for state mutations.

{
  "span_id": "agent.critic",
  "events": [
    {
      "name": "scratchpad.update",
      "timestamp": "2024-05-01T10:00:03Z",
      "attributes": {
        "key": "hypothesis",
        "value": "user wants SQL, not pandas",
        "writer": "researcher"
      }
    }
  ]
}

This turns the trace into a causal log of the system’s reasoning, not just its I/O. It is essential when an agent later contradicts an earlier step—you need to see which write won. In a single call, the prompt is immutable; in multi-agent systems, the “prompt” for the final model is assembled from a dozen intermediate state changes.

Cost and token attribution across nested calls

Single-call billing is one line item. Multi-agent runs nest calls: a synthesizer may call a retriever (no tokens), then a model (tokens), then a validator model (more tokens). To attribute cost per user request, you must roll up token usage along the trace tree. A loop where the critic sends the output back for revision multiplies tokens in ways invisible to flat logging.

A gateway with per-token usage metering lets you query aggregated spans:

SELECT
  trace_id,
  SUM(CAST(attributes->>'prompt_tokens' AS INT)) AS total_prompt,
  SUM(CAST(attributes->>'completion_tokens' AS INT)) AS total_completion
FROM spans
WHERE name = 'chat.completion'
GROUP BY trace_id;

If you skip this, a “cheap” router that triggers three sequential corrections can silently cost 10x a direct call. Per-token attribution also exposes which agent in the graph is the dominant cost driver, guiding where to cache or downgrade.

Failure isolation and partial graphs

In single-call tracing, an error means the whole span fails. In multi-agent systems, one worker may raise while others return valid results. The trace must represent a partial graph: the coordinator span succeeds (it recovered via fallback), but the researcher span is marked error.

{
  "span_id": "agent.researcher",
  "status": "ERROR",
  "events": [{"name": "exception", "attributes": {"type": "TimeoutError"}}]
}

If your tracer discards incomplete traces, you lose the exact context that explains the degraded output. Conversely, marking the entire trace failed hides the successful paths that could be reused. You need a model that treats error as a node property, not a trace property.

Tradeoffs: instrumentation overhead vs debuggability

Adding a span per tool call, per state mutation, and per model resolution increases overhead. At high QPS, you will pay in latency and storage. The pragmatic tradeoff is head-based sampling for live traffic and full capture for flagged conversations or trace IDs that exhibit anomalies.

You also face a schema problem: OpenTelemetry’s span model is generic, but LLM-specific attributes (token counts, model aliases, cache hints) need a convention. Adopt the GenAI semantic conventions early, or you will end up with incompatible dashboards. The cost of retrofitting a schema across millions of stored spans is far higher than the minor friction of standardizing at start.

Takeaway: design for a graph, not a line

Multi-agent tracing challenges are fundamentally about cardinality and causality. You are not tracing a call; you are tracing a distributed system where the nodes happen to be LLMs and the edges are prompts. Use a tracer that supports DAGs, propagate context across async boundaries, record resolved models and fallback events, emit state-change events, and meter tokens per span. Retrofitting a single-call APM will leave you blind at the exact moments your system behaves unexpectedly.

If you build with those requirements from day one, debugging a ten-agent workflow becomes as straightforward as reading a flame graph—not a forensic exercise.