OpenTelemetry collector configs for high-volume LLM traffic

Running an OpenTelemetry collector for LLM traffic at scale is different from instrumenting a typical microservice. Token-heavy requests, long generation latencies, and provider fallback paths produce telemetry volumes that will saturate a default pipeline within hours. This guide lays out a proven collector topology and config patterns to keep trace ingest costs and overhead under control while preserving the signals you actually need.

1. Split receive and process responsibilities

Don’t run a single collector that receives OTLP from apps, samples, redacts, and exports to your backend. At high request rates—think thousands of concurrent chat completions—that box becomes a bottleneck and a blast radius. Use a two-tier topology: lightweight agent collectors co-located with your inference services, and a centralized collector pool that does heavy processing.

The agent should do minimal work: receive, apply memory limit, compress, forward.

# agent-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp:
    endpoint: collector.internal:4317
    compression: zstd

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp]

The central collector then handles batching, sampling, and exports. This separation lets you scale the central pool independently and isolates redaction logic from the hot path of your model servers.

2. Batch aggressively but bound memory

LLM spans carry large attributes (prompts, completions). Exporting each span individually will hammer your backend and waste bandwidth. The batch processor is mandatory, but its defaults are conservative. Tune send_batch_size and timeout to match your load, and always pair it with memory_limiter.

# central-collector.yaml (excerpt)
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4000
    spike_limit_mib: 800
  batch:
    send_batch_size: 5000
    timeout: 10s
    send_batch_max_size: 10000

Pitfall: setting send_batch_size too high without raising memory_limiter limits causes OOM kills under traffic spikes. Conversely, a 1s timeout with size 500 forces too many small exports. Measure export request count in your metrics; aim for batches that keep export QPS below 10 per collector instance.

Tradeoff: larger batches increase tail latency for trace visibility. If you need near-real-time debugging, run a separate pipeline with lower batch timeout for error traces only.

3. Sample with context, not blind percentages

Head sampling (probabilistic at ingest) is cheap but throws away failed requests if you’re unlucky. For LLM apps, a failed generation or a provider timeout is the exact signal you cannot lose. Use tail sampling in the central collector to keep all errors and slow generations, while dropping most healthy traffic.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow_gen
        type: latency
        latency: {threshold_ms: 5000}
      - name: random_10pct
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

decision_wait must exceed your longest expected LLM call (some models stream for 30s+). num_traces caps memory; if you exceed it, oldest traces get dropped. In our experience, 50k concurrent traces needs ~2GiB RAM—size accordingly.

Pitfall: tail sampling is stateful. If you run multiple central collectors behind a load balancer without consistent trace ID hashing, a trace’s spans scatter and sampling decisions never complete. Use the loadbalancing exporter or route by trace ID at the agent.

4. Truncate and redact before export

A single 32k-token prompt embedded as a span attribute is a 128KB JSON blob. Multiply by 10k requests/min and your backend bill explodes. Use the transform processor to truncate known heavy fields and strip PII.

processors:
  transform:
    trace_statements:
      - context: span
        statements:
          - truncate(attributes["llm.prompt"], 1024)
          - truncate(attributes["llm.completion"], 1024)
          - delete(attributes["user.email"]) where attributes["user.email"] != nil
          - delete(attributes["authorization"]) where attributes["authorization"] != nil

Keep enough of the prompt to debug routing issues—first 1KB is usually sufficient. Never log full API keys; the delete statements should cover any accidental leakage from downstream SDKs.

Tradeoff: redaction reduces forensic value. If compliance requires full retention, ship unredacted spans to a separate secure backend via a second exporter, not your primary cheap store.

5. Derive token metrics from spans

Tracing alone won’t show token throughput or cost. The spanmetrics connector converts span durations and attributes into metrics. Pair it with a Prometheus exporter for dashboards.

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [0.1, 0.5, 1, 2, 5, 10, 30]
    dimensions:
      - name: llm.model
      - name: llm.provider

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [spanmetrics, otlp/backend]
    metrics:
      receivers: [spanmetrics]
      exporters: [prometheus]

Now you get request rate, error ratio, and latency per model without custom metric code. If you route through a gateway such as n4n.ai, the collector can ingest per-token metering and cache-control hints forwarded as span attributes, letting you build cost dashboards without custom instrumentation.

Pitfall: high-cardinality dimensions like llm.request.id will explode Prometheus. Only add dimensions that map to operational concerns (model, provider, status).

6. Preserve context across provider fallback

LLM gateways often retry or fall back to a different provider when rate-limited. Your instrumentation must propagate traceparent across those calls, and your collector must accept the resulting spans. Ensure the OTLP receiver accepts internal traffic from the gateway, and add a resource attribute to identify fallback hops.

processors:
  resource:
    attributes:
      - key: deployment.env
        value: prod
        action: upsert
      - key: collector.tier
        value: central
        action: upsert

If you use client routing directives (e.g., “prefer anthropic, fallback openai”), encode them as span events. The collector doesn’t need to parse them, but exporting them unaltered helps reconstruct the decision path.

Tradeoff: capturing every fallback attempt multiplies span count. Apply the tail sampling policy from step 3 so only anomalous fallbacks survive.

7. Scale out and monitor the collector itself

The collector is software; it fails. Deploy central collectors as a horizontally scaled deployment behind a load balancer. Use the loadbalancing exporter at the agent to hash by trace ID:

exporters:
  loadbalancing:
    protocol: otlp
    resolver:
      static:
        hostnames: [collector-0:4317, collector-1:4317, collector-2:4317]

Monitor collector health with its internal metrics (otelcol_processor_*, otelcol_exporter_*). Alert on exporter_send_failed_spans and memory_limiter.soft_limit_reclaimed.

Common pitfall: forgetting DNS TTL. If you use static hostnames, a collector pod IP change won’t be picked up. Use a headless service or a periodic resolver refresh.

What to do Monday morning

Stand up an agent collector beside one service. Ship truncated spans to a dev backend. Then add central batching and tail sampling. Only after that works, add spanmetrics and prometheus. Don’t boil the ocean—each layer adds observable behavior you’ll want to tune before the next.

The OpenTelemetry collector for LLM traffic is not a set-and-forget component. Treat its config as code, version it, and load-test with replayed traffic before peak.