Cross-service LLM span correlation with OpenTelemetry

Correlating LLM spans across microservices is the difference between guessing why a chat endpoint lagged and knowing that the summarizer worker burned four seconds on a retry. In a distributed system where a user request fans out to multiple model calls, a single W3C trace context is the only way to reconstruct the timeline.

1. Establish a single trace context at the edge

Most LLM apps start with an API gateway or BFF that receives the user request. Start a server span there and make sure every downstream HTTP call carries the traceparent header. OpenTelemetry’s propagators handle this if you wire them correctly.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.propagate import inject
import requests

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
RequestsInstrumentor().instrument()

tracer = trace.get_tracer(__name__)

def handle_request(user_id: str):
    with tracer.start_as_current_span("api.handle_chat") as span:
        span.set_attribute("user.id", user_id)
        headers = {}
        inject(headers)  # adds traceparent + tracestate
        requests.post("http://summarizer:8000/summarize", json={"text": "..."}, headers=headers)

If you skip inject, the summarizer starts a new trace. You’ve now lost the ability to correlate.

Edge pitfalls

Don’t rely on auto-instrumentation alone for non-HTTP transports. gRPC and Kafka need explicit context injection or you will silently orphan spans.

2. Instrument LLM client calls with semantic conventions

Inside each service, wrap the model call. Use the gen_ai semantic conventions so your spans show up in LLM-specific dashboards and can be queried uniformly.

from opentelemetry import trace
from openai import OpenAI

tracer = trace.get_tracer(__name__)
client = OpenAI()  # or any OpenAI-compatible base_url

def generate(prompt: str):
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", "gpt-4o-mini")
        span.set_attribute("gen_ai.request.prompt", prompt[:200])  # truncated
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
        )
        span.set_attribute("gen_ai.response.usage.total_tokens", resp.usage.total_tokens)
        return resp.choices[0].message.content

The truncation is deliberate. Storing full prompts in span attributes will blow up your backend storage and may leak PII. If you need the full prompt for debugging, write it to a blob store and put the URI in the span.

3. Propagate context through async workers

Microservices rarely stay synchronous. A request often drops a job on a queue and returns. The worker picks it up later. If you use Celery or RabbitMQ, the trace context dies unless you forward it.

# producer side
from opentelemetry.propagate import inject
headers = {}
inject(headers)
celery_app.send_task("workers.embed", args=[text], headers=headers)

# consumer side (Celery)
from opentelemetry.propagate import extract
from opentelemetry import trace

@celery_app.task
def embed(text):
    ctx = extract(self.request.headers)
    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("worker.embed", context=ctx) as span:
        # call embedding model
        ...

Without extract, the worker span is orphaned. Correlating LLM spans across microservices breaks exactly at this boundary because the async hop looks like a new root.

Tradeoff: baggage vs traceparent

Baggage lets you pass custom key-values (like user.tier) but increases header size. Use it sparingly; prefer trace context for hierarchy and logs for business data.

4. Keep the inference gateway inside the trace

If your architecture routes through an OpenAI-compatible inference gateway, the HTTP call must carry the same headers. For example, when you route through n4n.ai, pass the W3C trace headers on the request; it honors client routing directives and forwards provider cache-control hints, so the downstream provider span links into your trace instead of spawning a detached one.

from opentelemetry.propagate import inject
import requests

headers = {"Authorization": "Bearer <key>"}
inject(headers)
requests.post(
    "https://api.n4n.ai/v1/chat/completions",
    json={"model": "claude-3-5-sonnet", "messages": [...]},
    headers=headers,
)

If the gateway returns a provider-specific trace ID in a response header, log it as a span attribute (gen_ai.response.provider_trace_id) for cross-provider correlation.

5. Aggregate and query traces

Export spans to Jaeger, Tempo, or an OTel Collector. Structure your queries around the root service and then drill down.

# tempo query example
tempoquery --trace-id <hex> --org-id default

In Grafana, use the gen_ai.request.model attribute to group latency by model. You’ll quickly see which microservice burns the most tokens. Bridge metrics to traces by setting the same trace_id on your Prometheus exemplars—OpenTelemetry SDKs support this natively when you use the Meter alongside the Tracer.

Pitfall: high-cardinality attributes

Never set gen_ai.request.prompt to the full string. Never set user.email as a span attribute unless you have a retention policy that allows PII. These fields create unique index entries and inflate costs.

6. Common pitfalls and tradeoffs

Sampling. Head-based sampling drops traces before they reach the LLM call. If you need full visibility for debugging, use tail-based sampling at the collector, or set a tracestate flag from the edge to force-keep.

Context leakage. If you reuse a thread pool without clearing the active span, a later task may inherit the wrong parent. Always use start_as_current_span context managers.

Cost attribution. Token counts live on the LLM span. To attribute cost per user, propagate user.id via baggage and join in your metrics pipeline, not by parsing traces.

Retry storms. An LLM client retry spawns child spans. Make sure the retry span links to the original, or your latency view doubles. Set span.set_attribute("retry.count", n).

7. Verify correlation in tests

Use an in-memory exporter to assert parent-child relationships before shipping.

from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

exporter = InMemorySpanExporter()
# attach to provider as span processor
spans = exporter.get_finished_spans()
assert spans[0].parent is not None
assert spans[1].context.trace_id == spans[0].context.trace_id

This catches the most common bug: a missing inject or extract at a process boundary.

8. Minimal end-to-end checklist

1. Set TracerProvider at process start; export to collector.
2. Inject traceparent on every outbound call (HTTP, queue, worker).
3. Wrap model calls with gen_ai.* attributes, truncated inputs.
4. Pass headers to your inference gateway (e.g., OpenAI-compatible endpoint).
5. Extract context in async consumers.
6. Query by root trace ID; alert on gen_ai.response.usage.total_tokens per service.

Correlating LLM spans across microservices is not free: you pay with storage and a small latency overhead per span. But the first time a production incident spans three services and a model timeout, the trace graph pays for itself.