n4nAI
Open console →
Developer toolsTutorialOpenTelemetry Tracing for LLM Apps

Sending OpenTelemetry LLM traces to Grafana Tempo

Step-by-step tutorial to send OpenTelemetry LLM traces to Grafana Tempo: run Tempo with Docker, instrument a Python LLM client, query spans.

n4n Team2 min read498 words

Audio narration

Coming soon — every post will get a voice note here.

Getting end-to-end visibility into LLM latency, token spend, and provider fallbacks requires plumbing OpenTelemetry LLM traces to Grafana Tempo. This hands-on tutorial instruments a Python completion client with the OpenTelemetry SDK, exports spans over OTLP HTTP, and verifies them in a local Tempo instance.

Prerequisites

  • Python 3.11 or newer
  • Docker and Docker Compose installed
  • An API key for an OpenAI-compatible LLM endpoint
  • curl for quick HTTP checks
  • Basic familiarity with spans and trace context

1. Run Grafana Tempo and Grafana locally

Tempo accepts OTLP traces on both gRPC (4317) and HTTP (4318). We use HTTP to avoid protobuf tooling friction. The following docker-compose.yml starts Tempo with a local backend and a Grafana UI for searching.

version: "3.8"
services:
  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
    ports:
      - "3200:3200"   # Tempo query API
      - "4318:4318"   # OTLP HTTP receiver
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

Create tempo.yaml next to it:

server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318
storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo

Bring it up:

docker compose up -d
curl http://localhost:3200/ready
# expected output: "ready"

If you see ready, the receiver is bound and the query API is live.

2. Configure the OpenTelemetry SDK in Python

Install the minimal set of packages. We use the HTTP/protobuf exporter, not the gRPC one.

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http openai

Configure a TracerProvider with a BatchSpanProcessor pointed at the local OTLP HTTP endpoint. Set a service.name so Tempo can index the producer.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "llm-client",
    "service.version": "0.1.0",
})

provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("llm.tracer")

The BatchSpanProcessor buffers spans and sends them asynchronously. Call provider.shutdown() at process exit to flush.

3. Instrument an LLM call with GenAI attributes

Wrap the model call in a span and record the GenAI semantic conventions. If you route through n4n.ai, its OpenAI-compatible endpoint covers 240+ models with automatic fallback when a provider is degraded—set base_url to https://api.n4n.ai/v1 and use your key.

import openai
from opentelemetry.trace import Status, StatusCode

client = openai.OpenAI(
    api_key="YOUR_KEY",
    base_url="https://api.n4n.ai/v1",  # or "https://api.openai.com/v1"
)

with tracer.start_as_current_span("llm.completion") as span:
    span.set_attribute("gen_ai.system", "openai")
    span.set_attribute("gen_ai.request.model", "gpt-4o-mini")
    prompt = "Explain OTel traces in one sentence."
    span.set_attribute("gen_ai.prompt", prompt)

    try:
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
        )
        completion = resp.choices[0].message.content
        span.set_attribute("gen_ai.completion", completion)
        span.set_attribute("gen_ai.usage.prompt_tokens", resp.usage.prompt_tokens)
        span.set_attribute("gen_ai.usage.completion_tokens", resp.usage.completion_tokens)
        span.set_status(Status(StatusCode.OK))
    except Exception as e:
        span.record_exception(e)
        span.set_status(Status(StatusCode.ERROR))
        raise
    finally:
        print(f"Trace ID: {trace.format_trace_id(span.get_span_context().trace_id)}")

Run it:

python llm_trace.py
# Trace ID: 0x8a4c2f1b3c9d4e5f6a7b8c9d0e1f2a3b

The trace ID is the correlation key you’ll use in Tempo.

4. Confirm OpenTelemetry LLM traces to Grafana Tempo via API

Tempo’s query API accepts the trace ID (without 0x):

curl http://localhost:3200/api/traces/8a4c2f1b3c9d4e5f6a7b8c9d0e1f2a3b

A minimal successful response looks like:

{
  "traces": [
    {
      "traceID": "8a4c2f1b3c9d4e5f6a7b8c9d0e1f2a3b",
      "spans": [
        {
          "spanID": "1a2b3c4d5e6f7a8b",
          "operationName": "llm.completion",
          "tags": [
            {"key": "gen_ai.system", "value": "openai"},
            {"key": "gen_ai.request.model", "value": "gpt-4o-mini"},
            {"key": "gen_ai.prompt", "value": "Explain OTel traces in one sentence."},
            {"key": "gen_ai.usage.prompt_tokens", "value": 7}
          ]
        }
      ]
    }
  ]
}

If the response is {"traces":[]} or null, the exporter likely didn’t flush. Add provider.shutdown() at the end of the script and re-run.

5. Query OpenTelemetry LLM traces to Grafana Tempo in the UI

Point a browser at http://localhost:3000. Add a Tempo data source with URL http://tempo:3200. In Explore, pick Tempo and run a search with tag service.name=llm-client. The result list shows trace IDs; click one to see the waterfall.

Querying OpenTelemetry LLM traces to Grafana Tempo through the UI exposes the GenAI attributes as key-value pairs under each span. You can build dashboards that group by gen_ai.request.model to compare latency across providers, or alert on gen_ai.usage.completion_tokens spikes.

6. Propagate context across multiple calls

Real apps rarely make one call. Create child spans to track sequential or parallel requests:

with tracer.start_as_current_span("llm.pipeline") as root:
    with tracer.start_as_current_span("llm.embedding", context=trace.set_span_in_context(root)):
        # call embedding model, set gen_ai.* attrs
        pass
    with tracer.start_as_current_span("llm.completion", context=trace.set_span_in_context(root)):
        # call chat model
        pass

Always shut down the provider to flush buffered spans:

provider.shutdown()

Troubleshooting

  • Tempo returns empty: Verify the OTLP HTTP port mapping (4318:4318) and that the exporter endpoint includes /v1/traces.
  • Attributes missing in UI: Set them before the span closes; attributes added after span.end() are dropped.
  • Process exits before flush: BatchSpanProcessor is async. In short scripts, call provider.shutdown() explicitly.
  • Cardinality warnings: Full prompt text is high-cardinality. In production, store hashes or truncate to first 200 chars.

That pipeline gives you a reproducible path for OpenTelemetry LLM traces to Grafana Tempo, from instrumented Python code to a searchable trace backend.

Tagsopentelemetrygrafana-tempotracingtutorial

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsTracing a RAG pipeline end to end with OpenTelemetryLearn how to implement tracing RAG with OpenTelemetry in Python: instrument retrieval, augmentation, and generation steps with runnable code and spans.Developer toolsAdding OpenTelemetry to a Python LLM app in 20 minutesLearn to add OpenTelemetry to a Python LLM app in 20 minutes with this hands-on tutorial covering spans, attributes, and Jaeger output.Developer toolsCross-service LLM span correlation with OpenTelemetryA practical guide to correlating LLM spans across microservices with OpenTelemetry: propagate trace context, instrument clients, and avoid common pitfalls.Developer toolsDebugging slow LLM chains using OpenTelemetry waterfallsLearn how to trace and debug LLM chains with OpenTelemetry waterfalls. Step-by-step instrumentation, span analysis, and latency fixes for Python apps.Developer toolsExporting LLM traces to Jaeger with OpenTelemetryLearn how exporting LLM traces to Jaeger with OpenTelemetry works: stand up Jaeger, instrument Python LLM calls, and verify spans end to end.Developer toolsInstrumenting LLM calls with OpenTelemetry spansLearn how to implement instrumenting LLM calls with OpenTelemetry spans step by step, capturing latency, token usage, and errors across your LLM stack.

All opentelemetry tracing for llm apps posts →