Correlating traces across supervisor and worker agents

Correlating traces supervisor worker agents is the first thing you should wire up before adding a second worker to your orchestration layer. Without a shared context propagating from the supervisor down to each worker, you will spend hours guessing which subtask blew up the parent request instead of reading a single trace tree.

1. Establish a single correlation ID at the boundary

When an external request hits your supervisor agent, generate or accept a correlation ID immediately. This ID becomes the root trace identifier for everything downstream, regardless of how many workers the supervisor spawns. Do not rely on randomly generated IDs inside each worker; that severs the link before you start.

In Python, contextvars is the cleanest way to carry the ID without threading it through every function signature:

import contextvars

trace_id = contextvars.ContextVar("trace_id")
parent_span = contextvars.ContextVar("parent_span")

def handle_request(req_id: str):
    trace_id.set(req_id)
    supervisor_run()

If you run workers in separate threads, copy the context explicitly. contextvars.copy_context().run(target) preserves the values; bare threading.Thread(target=...) drops them. In async code, asyncio.create_task inherits the context automatically within the same event loop, but cross-process calls need serialization.

A common pitfall: reusing a server process across requests without resetting the var. Middleware should set trace_id per inbound call, not once at startup.

2. Propagate context through agent handoffs

Supervisors dispatch tasks via direct function calls, message queues, or RPC. For correlating traces supervisor worker agents across process boundaries, serialize the context into the message envelope. W3C traceparent is the interoperable standard, but a compact JSON blob is fine if you own both ends.

{
  "task": "summarize_logs",
  "payload": {"input": "..."},
  "trace_context": {
    "trace_id": "a1b2c3",
    "parent_span_id": "d4e5",
    "flags": 1
  }
}

The worker must reconstruct the context before executing any business logic. If you use a broker like RabbitMQ or Kafka, put trace_context in message headers, not the body, so it survives routing hops.

Tradeoff: full W3C propagation gives vendor-neutral spans and works with existing tracing backends, but adds header bloat and requires compliant parsers. A custom short ID is lighter and easier to log, but locks you to your own query layer. For most teams, adopting OTel’s format upfront saves a migration later.

3. Instrument each agent as its own span

Treat the supervisor and each worker as separate spans under one trace. A span is simply a timed operation with a parent reference. OpenTelemetry makes this explicit:

from opentelemetry import trace
tracer = trace.get_tracer("orchestrator")

with tracer.start_as_current_span("supervisor") as sup:
    ctx = trace.set_span_in_context(sup)
    with tracer.start_as_current_span("worker.translate", context=ctx):
        # worker logic here

If you are not ready for OTel, emit structured logs with trace_id, span_id, and parent_id fields. The hierarchy matters more than the framework. Each worker span should record at least: start time, end time, agent name, and status.

The classic mistake is creating a new trace at the worker entrypoint—calling tracer.start_span without a parent context. That produces orphan traces and defeats correlating traces supervisor worker agents. Always deserialize the parent and pass it in.

4. Capture LLM calls inside workers with model and token metadata

Workers in an LLM pipeline usually make model calls. Each call should be a nested span or log block with model name, prompt tokens, and completion tokens. This metadata is what turns “worker was slow” into “worker made three calls to a 70B model, burned 12k tokens, and one call hit a provider timeout.”

If you route through a gateway, capture its request identifier. When workers call models through n4n.ai, attach the gateway’s x-request-id to the span as an attribute; its per-token metering then lines up with your trace. That correlation lets you reconcile your internal span costs with the gateway invoice without manual CSV joins.

span.set_attribute("llm.model", "gpt-4o-mini")
span.set_attribute("llm.prompt_tokens", 1200)
span.set_attribute("gateway.request_id", response.headers.get("x-request-id"))

For streaming responses, open the span at first token and close on stream end; record token counts from the final usage chunk. Don’t emit a span per token—that floods the backend.

5. Aggregate traces in a backend that supports linkage

You need a system that stores the tree and lets you query by root. Jaeger, Grafana Tempo, or Honeycomb accept OTel natively. If you want to stay lean, push JSON logs to ClickHouse or Postgres with trace_id as a primary key column.

SELECT span_id, parent_id, operation, duration_ms
FROM spans
WHERE trace_id = 'a1b2c3'
ORDER BY start_ts;

Tradeoff: most tracing backends sample by default to control cost. For multi-agent debugging, use head-based 100% sampling in development and tail-based sampling in production that keeps all error or high-latency traces. Losing a healthy trace is fine; losing the one that failed is not.

If you use logs instead of a tracing UI, invest in a simple flame-graph renderer. A raw table of 200 spans is unreadable; a tree is not.

6. Handle dynamic worker spawning and retries

Supervisors often fan out to N workers. Each gets a child span under the same root. Retries must not create a new root; they are new spans with the same parent, otherwise you cannot see the retry storm that caused the slowdown.

for i, task in enumerate(tasks):
    with tracer.start_as_current_span(f"worker.{i}", context=parent_ctx):
        try:
            run_with_retry(task, max_attempts=3)
        except Exception as e:
            span.record_exception(e)
            raise

Pitfall: if a worker spawns sub-workers (agent calling agent), trace depth explodes. Set a max depth—say four levels—and flatten deeper calls into a single “nested-agent” span with a reference list. Otherwise your observability UI becomes a denial-of-service attack on your own engineers.

Also watch concurrency limits. A supervisor spawning 500 workers simultaneously will generate 500 spans in one trace; some backends cap span count per trace. Batch or cap fan-out.

7. Common pitfalls and tradeoffs

• Context leakage: long-lived servers that forget to reset trace_id between requests merge unrelated traces. Use per-request middleware.
• Over-instrumentation: spawning a span for every helper function floods the backend. Span at agent boundaries; log inside the worker.
• Privacy: trace payloads often contain user inputs sent to workers. Mask PII before export or you create a compliance incident.
• Clock skew: across machines, span timestamps drift. Run NTP and prefer recording relative durations where possible.
• Async context loss: the easiest way to break correlating traces supervisor worker agents is an await that drops the context because someone used loop.create_task outside the current context. Test with an artificial delay and verify the tree.
• Sampling blind spots: if you sample at 1% in prod, you will never debug the rare worker deadlock. Use tail sampling on failures.

8. Minimal reference implementation

Below is a skeleton you can paste into a service to see the shape of linked spans without external dependencies. It uses contextvars and manual IDs, printing JSON lines you can pipe to any log store.

import uuid, time, json

class Span:
    def __init__(self, name, trace_id, parent_id=None):
        self.id = uuid.uuid4().hex[:8]
        self.trace_id = trace_id
        self.parent_id = parent_id
        self.name = name
        self.start = time.time()
    def end(self):
        self.duration = time.time() - self.start
        print(json.dumps({"trace_id": self.trace_id, "span_id": self.id,
                          "parent_id": self.parent_id, "name": self.name,
                          "dur_ms": round(self.duration*1000, 1)}))

def supervisor(req_id):
    root = Span("supervisor", req_id)
    for w in ["fetch", "parse", "summarize"]:
        worker(req_id, root.id, w)
    root.end()

def worker(trace_id, parent_id, task):
    s = Span(f"worker.{task}", trace_id, parent_id)
    time.sleep(0.01)  # pretend work
    s.end()

supervisor("req-123")

Run it; you get three child spans sharing req-123 as trace_id and pointing at the supervisor’s span_id. Replace the print with an OTel exporter or a log shipper and you have production-ready correlation.

Correlating traces supervisor worker agents is not glamorous, but it is the backbone of debuggable multi-agent systems. Build the propagation path first, instrument at boundaries, and only then scale the worker pool.