Tracing agent-to-agent handoffs in multi-agent systems

Tracing agent-to-agent handoffs is the difference between a multi-agent system you can debug and one that silently corrupts state. When an orchestrator delegates to a sub-agent, the handoff carries context, tool constraints, and model routing decisions that must be observable end to end.

Step 1: Establish a shared trace context

Every handoff needs a correlation identifier that survives across process boundaries. Use the W3C traceparent header or an explicit correlation_id field in your agent message envelope. Don’t rely on timestamps or log scraping to reconstruct order.

In a Python multi-agent loop, inject the context at the call site:

from opentelemetry import trace
from opentelemetry.propagate import inject

def dispatch_to_agent(agent_name: str, payload: dict):
    ctx = trace.get_current_span().get_span_context()
    headers = {}
    inject(headers)  # populates traceparent
    # pass headers with your RPC / HTTP call
    return http.post(f"/agents/{agent_name}", json=payload, headers=headers)

The receiving agent must extract the context before creating its own span. This makes the child span correctly parented.

If you use a message broker (Redis, Kafka, SQS), serialize the traceparent into the message metadata. For Kafka, set the header traceparent as a byte string. For SQS, put it in MessageAttributes. Skipping this step is the most common reason tracing agent-to-agent handoffs fails in async systems.

Context in async loops

When using asyncio, the current span is stored in contextvars. Ensure your dispatcher runs inside the same task. If you spawn a new thread, use contextvars.copy_context() and run the agent within it.

Step 2: Instrument each agent as a distinct span

Wrap agent entry points with a span that records the agent name, input schema version, and model identifier. OpenTelemetry’s SDK gives you this with minimal boilerplate.

from opentelemetry import trace
tracer = trace.get_tracer("multi_agent.system")

def agent_entrypoint(agent_name, payload, headers):
    ctx = extract(headers)  # opentelemetry.propagate.extract
    with tracer.start_as_current_span(f"agent.{agent_name}", context=ctx) as span:
        span.set_attribute("agent.name", agent_name)
        span.set_attribute("agent.input_keys", ",".join(payload.keys()))
        result = run_agent(agent_name, payload)
        span.set_attribute("agent.output_tokens", result["usage"]["completion_tokens"])
        return result

Without this, tracing agent-to-agent handoffs becomes a guess about which log line belongs to which invocation.

Sampling strategy

Default OTel samplers keep everything, which is fine for dev. In production, use a ParentBased sampler so handoffs are always captured if the root is sampled. Missing a child because of independent sampling breaks the tree.

Step 3: Record handoff metadata explicitly

A handoff is a contract: the caller promises a shape, the callee returns a shape. Emit those as span events so you can reconstruct the exact payload without full body logging.

span.add_event("handoff.request", attributes={
    "target_agent": "retriever",
    "inputs": json.dumps({"query": payload["q"]}),
    "model": payload.get("model", "default")
})

On the receiving side, log the acceptance:

span.add_event("handoff.accept", attributes={
    "source_agent": "orchestrator",
    "received_keys": ",".join(payload.keys())
})

This event pair is the atomic unit for tracing agent-to-agent handoffs. When something breaks, you see the request event in the parent and the missing accept in the child.

Redact before logging

Never put raw user PII in span events. Hash or truncate sensitive fields. The trace is only useful if you can store it compliantly.

Step 4: Propagate model routing and fallback signals

If your agents call LLMs through a gateway, the model selection and any fallback are part of the handoff semantics. For example, routing a sub-agent to a specific provider because of cost or latency is a decision that must be visible.

When you route through n4n.ai, the gateway honors client routing directives and forwards provider cache-control hints. Capture those as span attributes so the trace shows why a particular model answered.

span.set_attribute("llm.route.directive", payload.get("route", "auto"))
span.set_attribute("llm.cache_control", headers.get("x-cache-control", "none"))

If a provider is degraded, the automatic fallback will switch models; record the resolved model from the response, not the requested one.

resolved = result["model"]  # e.g. "anthropic/claude-3.5" after fallback
span.set_attribute("llm.resolved_model", resolved)

Add the token usage from the gateway’s per-token metering to the span so cost can be attributed per handoff.

Step 5: Aggregate and query traces centrally

Spans sitting in a single process are useless for a distributed handoff. Run an OpenTelemetry collector and a backend like Jaeger.

docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest

Configure your exporter:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317")))
trace.set_tracer_provider(provider)

Now a handoff from orchestrator to retriever to summarizer appears as a single trace tree in the Jaeger UI. Use the agent.name attribute as a filter to isolate a specific path.

Retention

Set a TTL on your trace backend. Handoff traces for debugging rarely need to live beyond 7 days. Aggregate metrics (count of handoffs, p95 latency) can live longer.

Step 6: Verify a handoff end to end

Write a test that forces a handoff and asserts the span hierarchy. This catches context propagation regressions before they hit production.

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, InMemorySpanExporter

def test_handoff_trace():
    exporter = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    # simulate orchestrator -> sub-agent
    with tracer.start_as_current_span("agent.orchestrator") as parent:
        dispatch_to_agent("retriever", {"q": "test"})  # uses current ctx

    spans = exporter.get_finished_spans()
    names = [s.name for s in spans]
    assert "agent.orchestrator" in names
    assert "agent.retriever" in names
    parent_id = [s.context.span_id for s in spans if s.name == "agent.orchestrator"][0]
    child = [s for s in spans if s.name == "agent.retriever"][0]
    assert child.parent.span_id == parent_id

Run it with pytest. If the assertion on child.parent.span_id fails, your propagation injection is broken.

Live verification

Deploy the instrumented agents to a staging environment and trigger a real multi-agent task. Open Jaeger, find the trace by correlation_id, and confirm you see the request/accept events. That is the final proof tracing agent-to-agent handoffs works.

Verification checklist

After following these steps, confirm success:

A handoff from agent A to agent B shows a single trace with correct parent-child spans in Jaeger.
Each handoff request/accept event pair appears with matching target_agent and source_agent attributes.
Model routing directives and resolved models are present on spans that made LLM calls.
Your CI test for span hierarchy passes consistently.
A staging run displays the full handoff chain within one trace ID.

Tracing agent-to-agent handoffs is not optional once you run more than two agents in production. The above pipeline turns an opaque message queue into a debuggable graph.