n4nAI
Open console →
Developer toolsHow-toOpenTelemetry Tracing for LLM Apps

Measuring token counts as OpenTelemetry span attributes

Learn how to emit token counts as OpenTelemetry span attributes for LLM apps, with runnable Python code for instrumentation and verification.

n4n Team3 min read645 words

Audio narration

Coming soon — every post will get a voice note here.

Most LLM apps treat token usage as a billing afterthought, logged to stdout and forgotten. Recording token counts as OpenTelemetry span attributes turns that data into first-class telemetry you can query per request, per user, or per model. This guide walks through instrumenting a Python service to capture prompt, completion, and total tokens on each LLM call span.

Step 1: Install and initialize the OpenTelemetry SDK

Start with the OTel API, SDK, an OTLP exporter, and the OpenAI client. We use the gRPC OTLP exporter because most collectors listen on 4317.

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc openai

Initialize a TracerProvider once at process startup. In production you point the exporter at your collector. For local verification, swap in a ConsoleSpanExporter (shown in Step 5).

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
otlp = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(otlp))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("llm.client")

Keep the tracer module-level. Creating a new provider per request leaks span processors and corrupts context.

Step 2: Capture token usage from the LLM response

OpenAI-compatible APIs return a usage object on every completion. The shape is stable: prompt_tokens, completion_tokens, total_tokens. If you front your calls with an OpenAI-compatible gateway such as n4n.ai, the response still carries the standard usage object, and the gateway handles per-token usage metering and fallback across providers behind one endpoint.

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain span attributes."}],
)

u = resp.usage
print(u.prompt_tokens, u.completion_tokens, u.total_tokens)

Do not compute token counts client-side with a tokenizer. Provider-side counts include exact prompt caching, special tokens, and reasoning overhead that local tokenizers miss. Always trust the usage field.

Step 3: Record token counts as OpenTelemetry span attributes

Wrap the LLM call in a span and set attributes. Use flat, dot-namespaced keys. OTel span attributes must be strings, ints, floats, bools, or arrays of those—token counts are ints, so they pass through cleanly.

def chat_with_telemetry(messages, model="gpt-4o-mini", user_id="anon"):
    with tracer.start_as_current_span("llm.chat_completion") as span:
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.user_id", user_id)

        resp = client.chat.completions.create(model=model, messages=messages)
        u = resp.usage

        # token counts as OpenTelemetry span attributes
        span.set_attribute("llm.token_count.prompt", u.prompt_tokens)
        span.set_attribute("llm.token_count.completion", u.completion_tokens)
        span.set_attribute("llm.token_count.total", u.total_tokens)

        return resp

Avoid putting the full prompt or completion text into span attributes by default—that bloats the trace store and often leaks PII. If you need payloads, use span events or a separate log stream with the same trace ID.

Step 4: Add routing and cache context

Real systems route across models and use provider caching. Extend the span with the resolved model and cache hit status. OpenAI-compatible responses expose prompt_tokens_details.cached_tokens on some models; surface it.

def chat_with_telemetry(messages, model="gpt-4o-mini", user_id="anon"):
    with tracer.start_as_current_span("llm.chat_completion") as span:
        span.set_attribute("llm.requested_model", model)
        span.set_attribute("llm.user_id", user_id)

        resp = client.chat.completions.create(
            model=model,
            messages=messages,
            extra_body={"cache_control": {"type": "ephemeral"}},
        )
        u = resp.usage
        span.set_attribute("llm.resolved_model", resp.model)
        span.set_attribute("llm.token_count.prompt", u.prompt_tokens)
        span.set_attribute("llm.token_count.completion", u.completion_tokens)
        span.set_attribute("llm.token_count.total", u.total_tokens)

        cached = getattr(u.prompt_tokens_details, "cached_tokens", 0) or 0
        span.set_attribute("llm.token_count.cached", cached)
        return resp

Gateways that honor client routing directives will return the actually-served model in resp.model. Recording both requested and resolved model makes fallback visible in traces.

Step 5: Export and verify span attributes

For a quick local check, replace the OTLP exporter with ConsoleSpanExporter. You will see the attributes printed as JSON when the span flushes.

from opentelemetry.sdk.trace.export import ConsoleSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

Run your script:

python llm_client.py

Look for output containing:

{
  "name": "llm.chat_completion",
  "attributes": {
    "llm.model": "gpt-4o-mini",
    "llm.token_count.prompt": 12,
    "llm.token_count.completion": 34,
    "llm.token_count.total": 46,
    "llm.token_count.cached": 0
  }
}

For end-to-end verification against a real collector, start Jaeger all-in-one:

docker run -d --name jaeger -p 4317:4317 -p 16686:16686 jaegertracing/all-in-one:latest

Point the OTLP exporter at localhost:4317, run the script, then open http://localhost:16686. Find the llm.chat_completion span and confirm the token attributes appear in the tags section. If they are missing, check that the span closed (the with block exited) before process exit—BatchSpanProcessor needs a moment to flush, or call provider.shutdown() at the end.

Step 6: Turn attributes into metrics

Spans are great for per-request debugging, but dashboards want metrics. Use the OTel Collector spanmetrics processor or a tail-sampling processor to roll up llm.token_count.total by llm.resolved_model. A minimal collector config snippet:

processors:
  spanmetrics:
    metrics:
      - name: llm_tokens_total
        type: Sum
        unit: tokens
        attributes: [llm.resolved_model]
        span_attribute: llm.token_count.total

This generates a cumulative counter without writing extra instrumentation code. You avoid double-counting by relying on the same token counts as OpenTelemetry span attributes that already exist on every call.

Pitfalls to avoid

Never set token counts as strings ("46"). Some backends will drop non-numeric types or break aggregation. Keep attribute keys low-cardinality: llm.user_id is fine if you have thousands of users, but llm.request_id as an attribute is not—put that in the span name or a span event instead.

If you batch multiple LLM calls inside one logical operation, create a parent span and child spans per call. Summing children in a backend query is straightforward; splitting a single span’s attributes after the fact is not.

Recording token counts as OpenTelemetry span attributes is a small change that pays off the first time you need to explain why a specific user’s bill spiked or why a model switch regressed latency. Ship it before you need it.

Tagsopentelemetrytoken-usagespan-attributestracing

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsCross-service LLM span correlation with OpenTelemetryA practical guide to correlating LLM spans across microservices with OpenTelemetry: propagate trace context, instrument clients, and avoid common pitfalls.Developer toolsDebugging slow LLM chains using OpenTelemetry waterfallsLearn how to trace and debug LLM chains with OpenTelemetry waterfalls. Step-by-step instrumentation, span analysis, and latency fixes for Python apps.Developer toolsExporting LLM traces to Jaeger with OpenTelemetryLearn how exporting LLM traces to Jaeger with OpenTelemetry works: stand up Jaeger, instrument Python LLM calls, and verify spans end to end.Developer toolsInstrumenting LLM calls with OpenTelemetry spansLearn how to implement instrumenting LLM calls with OpenTelemetry spans step by step, capturing latency, token usage, and errors across your LLM stack.Developer toolsOpenTelemetry context propagation for async LLM callsLearn how to implement OpenTelemetry context propagation async patterns for LLM apps, keeping traces intact across asyncio tasks and threads.Developer toolsOpenTelemetry semantic conventions for GenAI, explainedA precise explainer of OpenTelemetry GenAI semantic conventions: what they define, how spans model LLM calls, and why standardized tracing matters.

All opentelemetry tracing for llm apps posts →