Most teams track LLM calls through three disconnected lenses: APM traces for latency, provider JSON for token counts, and a finance spreadsheet for cost. A unified LLM logging schema collapses these into a single structured event, so every request emits one record with timing, usage, and price attached. Build this once and you get per-request cost attribution and latency breakdowns without gluing dashboards together.
Why scattered LLM telemetry fails
When latency lives in Datadog, token counts in OpenAI’s response JSON, and cost in a monthly CSV, you cannot answer “which endpoint burned $50 yesterday?” without a join across systems that don’t share keys. The join key is usually missing: APM spans use trace IDs, provider responses use request IDs, and finance uses date buckets.
Worse, LLM calls fail partially. A stream may disconnect after 200 tokens. If you only log on success, you lose the expensive half-finished requests. A unified LLM logging schema forces you to decide up front what a “request event” means, even when it errors.
The tradeoff is schema rigidity. You will need to add fields as new providers expose new metadata (cache tokens, reasoning steps). Accept that and version the schema.
Core fields of a unified LLM logging schema
Start with a minimal but complete event. Below is a JSON shape we ship in production. It is intentionally flat where possible to keep indexing cheap.
{
"schema_version": 1,
"timestamp": "2024-05-12T18:22:01.123Z",
"trace_id": "req-8f2c1a",
"model": "gpt-4o-mini",
"provider": "openai",
"route_directive": "prefer:openai,fallback:anthropic",
"latency_ms": {
"time_to_first_token": 320,
"total": 1450
},
"tokens": {
"prompt": 1200,
"completion": 350,
"cache_read": 800,
"cache_write": 0
},
"cost_usd": 0.0021,
"error": null,
"finish_reason": "stop"
}Field notes:
trace_idshould match your application trace, not the provider’s. Correlation is the whole point.modelis the logical model you called. If the provider aliases it (e.g.,gpt-4->gpt-4-0613), log both or log the resolved one inprovider_model.latency_mssplits time-to-first-token (TTFT) from total. TTFT is the user-perceived “is it stuck?” metric for streaming UIs.tokens.cache_readandcache_writeare separate because they bill at different rates.cost_usdis a float, but store milli-cents if you need exact aggregation.
Measure latency at the client boundary
Do not trust server-reported latency alone. Network egress and client serialization add real milliseconds. Wrap your client so the clock starts before the HTTP call and stops after you parse the last byte.
For streaming, TTFT is the delta between request start and the first chunk with content. Total latency includes stream closure.
import time, openai, json
client = openai.OpenAI()
def complete(prompt, trace_id):
start = time.monotonic()
first_token_ts = None
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
stream=True,
stream_options={"include_usage": True}
)
final_chunk = None
for chunk in stream:
if first_token_ts is None and chunk.choices and chunk.choices[0].delta.content:
first_token_ts = time.monotonic()
final_chunk = chunk
total_ms = (time.monotonic() - start) * 1000
ttft_ms = (first_token_ts - start) * 1000 if first_token_ts else None
usage = final_chunk.usage
event = {
"schema_version": 1,
"trace_id": trace_id,
"model": "gpt-4o-mini",
"latency_ms": {"time_to_first_token": ttft_ms, "total": total_ms},
"tokens": {
"prompt": usage.prompt_tokens,
"completion": usage.completion_tokens,
"cache_read": getattr(usage, "prompt_tokens_details", {}).get("cached_tokens", 0)
}
}
return eventPitfall: time.monotonic() is per-process. If you run multiple workers, align on UTC timestamps for cross-service events. Another: if the stream throws mid-way, catch the exception and emit an event with error set and the tokens you counted so far.
Attribute token usage and cost accurately
Providers return a usage object. Map it directly, but normalize field names. Anthropic calls it input_tokens; OpenAI calls it prompt_tokens. Write one adapter per provider.
Cost is where teams slip. You can maintain a price table keyed by model and token type, but cache discounts and dynamic pricing make it stale within weeks. If you route through a gateway such as n4n.ai, the response includes per-token usage metering, so cost_usd arrives precomputed and reflects cache discounts. If you call providers directly, compute cost in a single pure function and unit-test it against three known invoices.
PRICING = {
"gpt-4o-mini": {"prompt": 0.00015, "completion": 0.0006, "cache_read": 0.000015}
}
def compute_cost(model, tokens):
p = PRICING[model]
return (tokens["prompt"] - tokens.get("cache_read",0)) * p["prompt"] \
+ tokens.get("cache_read",0) * p["cache_read"] \
+ tokens["completion"] * p["completion"]Tradeoff: precomputed cost from a gateway saves maintenance but couples your logs to that gateway’s price schedule. Direct computation keeps you independent but wrong if you miss a price change.
Handle streaming, partial failures, and fallback
A unified LLM logging schema must encode what actually happened, not what you intended. If your client sent route_directive: prefer:openai,fallback:anthropic and OpenAI was rate-limited, log provider: anthropic and fallback_triggered: true. This is the only way to measure fallback rate.
For errors, set error to a short code (rate_limit, timeout, context_length) and still record tokens.prompt if the request reached the provider. Never omit the event because it failed; those are your most expensive lessons.
Emit logs without blocking the request path
Logging must be async. A synchronous print or HTTP POST to your log sink adds tail latency to user requests. Use an in-process queue drained by a worker, or a local stdout JSON pipe collected by Fluent Bit.
import asyncio, json
log_queue = asyncio.Queue(maxsize=1000)
async def log_worker():
while True:
event = await log_queue.get()
# replace with Kafka/OTLP send
print(json.dumps(event))
log_queue.task_done()
async def emit(event):
try:
await log_queue.put(event)
except asyncio.QueueFull:
# drop or metric++ ; never block the request
passIf you batch, keep batch size small (<100 events) to limit data loss on crash.
Version the schema from day one
Add schema_version as an integer. When you add reasoning_tokens next quarter, bump to 2 and keep the ingestion mapper able to handle both. Write your dashboards to filter on schema_version >= 1 so old events still count.
Query example (generic SQL on a JSON column):
SELECT model,
AVG((latency_ms->>'total')::float) AS avg_total_ms,
SUM((tokens->>'completion')::int) AS total_completion_tokens,
SUM(cost_usd) AS spend
FROM llm_events
WHERE timestamp > now() - interval '24 hours'
GROUP BY model;Common pitfalls we keep seeing
- Mixing units. One service logs seconds, another milliseconds. Pick milliseconds and enforce in CI with a schema linter.
- Ignoring cache tokens. If you count
cache_readas plain prompt tokens, your cost is 10x wrong. - Logging model alias only.
gpt-4is not a billable SKU; resolve it. - No fallback flag. You think you are 100% on the cheap provider, but you are silently paying premium fallback rates.
- Synchronous logging. You added 40ms p99 because you POSTed to Splunk inline.
Deployment checklist
- Define the event JSON with the fields above and a
schema_version. - Write a client wrapper that captures TTFT and total latency via
time.monotonic(). - Map provider
usageto your token fields, including cache details. - Compute or receive
cost_usd; if using a gateway with metering, trust its number but log the raw token counts too. - Emit events through an async queue; never block the request thread.
- Add
fallback_triggeredanderrorpaths for every failure mode you have hit in the last month. - Set up one daily query that sums
cost_usdbymodeland alerts on anomalies.
A unified LLM logging schema is not a luxury; it is the difference between guessing and knowing where your latency and dollars go. Ship the first version today and extend it when the next provider quirk appears.