Debugging concurrent workflows gets painful fast. A trace timeline for parallel agent execution turns interleaved LLM calls, tool invocations, and handoffs into a single queryable view instead of scattered logs. Build it once and you can see exactly where agents block, where a provider stalls, and which branch wasted tokens.
Step 1: Define the span schema before writing instrumentation
A usable trace timeline for parallel agent execution starts with a fixed schema. You are not just logging; you are building a directed acyclic graph of spans under one trace ID. Each agent run is a root span. Every LLM call, tool use, or sub-agent handoff is a child span.
Decide on attributes up front:
agent.id— stable identifier for the workeragent.role— planner, retriever, critic, etc.llm.model— resolved model stringllm.tokens— total tokens consumederror— boolean or empty
A minimal span object looks like this:
{
"trace_id": "4b7e1c2a",
"span_id": "a1",
"parent_id": null,
"name": "agent.planner",
"start": 1718200000000,
"end": 1718200001200,
"attributes": {
"agent.id": "planner-0",
"agent.role": "planner"
}
}If you skip this step, you will end up with spans that cannot be joined cleanly in the query layer.
Step 2: Initialize OpenTelemetry with a batch exporter
Use the OpenTelemetry Python SDK. A BatchSpanProcessor avoids blocking your agent loop on every span flush. Point the OTLP exporter at a local collector (Jaeger, Tempo, or OTel Collector).
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
provider = TracerProvider()
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("multi_agent_orchestrator")Run the collector before your app:
docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latestStep 3: Wrap each agent in a root span and propagate context
Asyncio tasks do not automatically share the active span context. You must capture the context from the orchestrator and pass it explicitly to each task. Otherwise every agent span becomes its own trace.
import asyncio
from opentelemetry import trace
async def run_agent(agent_id, task, parent_ctx):
with tracer.start_as_current_span(f"agent.{agent_id}", context=parent_ctx) as span:
span.set_attribute("agent.id", agent_id)
span.set_attribute("agent.task", task)
# simulated planning + tool call
await asyncio.sleep(0.05)
with tracer.start_as_current_span("tool.search") as tool_span:
tool_span.set_attribute("tool.name", "vector_query")
await asyncio.sleep(0.02)
async def orchestrate(tasks):
root = tracer.start_span("orchestration")
ctx = trace.set_span_in_context(root)
workers = [asyncio.create_task(run_agent(i, t, ctx)) for i, t in enumerate(tasks)]
await asyncio.gather(*workers)
root.end()This guarantees all agent spans share trace_id and can be rendered on one timeline.
Step 4: Record LLM calls as nested spans with routing metadata
Each LLM call should be a child span of the agent that made it. If you route through a gateway, attach the resolved model and token count as attributes. Routing through n4n.ai gives one OpenAI-compatible endpoint for 240+ models with automatic fallback when a provider is degraded; the gateway forwards cache-control hints and meters per-token usage, which we attach as span attributes.
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
async def call_llm(span, prompt):
with tracer.start_as_current_span("llm.call") as llm_span:
resp = await client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": prompt}],
extra_headers={"x-n4n-router": "auto"}
)
llm_span.set_attribute("llm.model", resp.model)
llm_span.set_attribute("llm.tokens", resp.usage.total_tokens)
llm_span.set_attribute("llm.prompt_tokens", resp.usage.prompt_tokens)
span.set_attribute("agent.llm_model", resp.model)Now the timeline shows exactly which agent called which model, how long it waited, and what it cost in tokens.
Step 5: Aggregate parallel spans into a timeline view
The trace timeline for parallel agent execution is only useful if you can see overlap. Pull the trace from Jaeger’s HTTP API and flatten spans into start/duration rows.
import requests
def fetch_timeline(trace_id):
r = requests.get(f"http://localhost:16686/api/traces/{trace_id}")
payload = r.json()
spans = payload["data"][0]["spans"]
rows = []
for s in spans:
rows.append({
"name": s["operationName"],
"start": s["startTime"],
"duration": s["duration"],
"attrs": {k["key"]: k["value"] for k in s.get("tags", [])}
})
return sorted(rows, key=lambda x: x["start"])
def print_gantt(rows):
for r in rows:
bars = int(r["duration"] / 50)
print(f"{r['name']:<20} |{'-' * bars}")For a production UI, feed the same rows to a React component using vis-timeline or plotly Gantt. The key is that the x-axis is absolute epoch microseconds, so concurrent agents line up horizontally.
Step 6: Verify success with a concurrency test
You need proof that the timeline reflects true parallelism, not sequential emulation. Launch three agents simultaneously, then assert their root spans overlap and share a trace ID.
import pytest
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
def test_parallel_overlap():
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
global tracer
tracer = trace.get_tracer("test")
trace_id = "test-trace-1"
root = tracer.start_span("orchestration", trace_id=trace.TraceId.from_string(trace_id.zfill(32)))
ctx = trace.set_span_in_context(root)
asyncio.run(orchestrate(["task-a", "task-b", "task-c"]))
root.end()
rows = fetch_timeline(trace_id)
agents = [r for r in rows if r["name"].startswith("agent.")]
assert len(agents) == 3
starts = [r["start"] for r in agents]
ends = [r["start"] + r["duration"] for r in agents]
# overlap exists if latest start is before earliest end
assert max(starts) < min(ends)Run it:
pytest test_timeline.py -qIf the test passes and the Gantt output shows three agent.* rows overlapping with nested llm.call bars, the trace timeline for parallel agent execution is working. You can now spot a slow provider, a stuck retriever, or an agent that silently retried without guessing from logs.
Keep the schema strict, export in batches, and never let an agent spawn a span without a parent context. That discipline is what makes the timeline trustworthy when you scale to a dozen concurrent workers.