A usable JSON logging schema for LLM requests is the difference between debugging latency spikes in minutes and guessing for hours. This guide lays out a concrete, extensible JSON logging schema for LLM requests that captures the fields you actually need for cost, latency, and quality analysis.
1. Anchor every log on a request ID and timestamps
Generate a unique request_id at the edge of your system, before you call any model. Propagate it through to the LLM gateway and back. Use UTC RFC3339 timestamps for started_at and ended_at; compute duration_ms server-side with a monotonic clock, not from the provider’s reported time alone.
{
"request_id": "req_01h9x8n3k2",
"trace_id": "trace_88a",
"started_at": "2024-05-12T18:22:01.123Z",
"ended_at": "2024-05-12T18:22:04.987Z",
"duration_ms": 3864
}Never rely on client clock skew. If you batch, log each sub-request with the same trace_id but distinct request_id. In async flows, generate the ID at submission time so you can correlate timeouts that never produce a response object.
2. Capture model and routing metadata
Record the model you intended to call (requested_model) and the model that actually served the response (served_model). They diverge when you use a gateway with fallback or aliases. Include the provider endpoint and any routing hints you sent.
{
"requested_model": "gpt-4o",
"served_model": "gpt-4o-2024-05-13",
"provider": "openai",
"endpoint": "https://api.openai.com/v1/chat/completions",
"routing_directive": "prefer-cache"
}If you sit behind a gateway that performs automatic fallback when a provider is rate-limited, log the actually-served provider and the fallback chain. For example, n4n.ai exposes the final provider and honors client routing directives, so capture served_provider from the response headers or body. Version pinning matters: a model alias like gpt-4o can resolve to different weight versions week to week. Log the resolved version always.
3. Record token usage as discrete integers
Aggregate token counts must be top-level integers, not nested strings. Split prompt, completion, and total. If the provider returns cached token counts, log them separately; they change your cost math.
{
"tokens": {
"prompt": 412,
"completion": 188,
"total": 600,
"cached_prompt": 256
}
}Do not compute cost in the log emitter. Log raw usage and join with price tables at analysis time. Prices change; your logs shouldn’t lie retroactively. If a provider returns reasoning or system tokens as separate line items, add them as explicit fields rather than folding them into prompt.
4. Reference prompts and responses, don’t embed them
Full prompt and completion bodies blow up your log volume and leak PII. Store a content hash and a pointer to an object store or replay buffer.
{
"prompt_ref": "s3://llm-logs/req_01h9x8n3k2/prompt.json",
"response_ref": "s3://llm-logs/req_01h9x8n3k2/response.json",
"prompt_hash": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
"response_hash": "sha256:2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae"
}For low-traffic internal tools, embedding truncated text under prompt_excerpt (first 200 chars) is fine. Make the truncation explicit with excerpt_chars. Never log full secrets, API keys, or unredacted user identifiers in the hot path.
5. Track finish reasons and truncation
The finish_reason tells you if the model stopped naturally, hit a limit, or was filtered. Log it as a string enum, and add truncated boolean for client-side caps.
{
"finish_reason": "length",
"truncated": true,
"max_tokens_requested": 1024,
"stop_sequences": ["\n\n"]
}Common enum values: stop, length, content_filter, tool_calls, error. Missing this field is the classic cause of “why is the output short?” tickets. If you use streaming, only log the aggregated finish reason after the stream closes.
6. Log errors with structured context
On failure, emit the same base schema plus an error object. Include HTTP status, provider error code, and a retry count. Never log secrets from the response.
{
"request_id": "req_01h9x8n3k2",
"error": {
"type": "rate_limit",
"provider_status": 429,
"code": "rate_limit_exceeded",
"retries": 2,
"message": "Rate limit reached for requests"
}
}Distinguish between a timeout (no provider response) and a 5xx (provider returned error). Your fallback logic depends on that signal. Add fallback_attempted: true if your client or gateway tried another provider.
7. Add client and application tags
Tag logs with the calling service, environment, and user tier. Keep these as flat string fields to make indexing trivial.
{
"app": "support-bot",
"env": "prod",
"user_tier": "pro",
"feature": "summarize-thread"
}Avoid high-cardinality fields like user_id in the hot log path; ship those to a separate audit stream. High cardinality destroys index performance in Elasticsearch and BigQuery alike.
8. Emit one JSON object per line
Use newline-delimited JSON (NDJSON). One request equals exactly one line. This lets you pipe to jq or load into columnar stores without a custom parser.
import json, logging, time
def log_llm_request(record: dict):
# record already contains the schema fields
logging.info(json.dumps(record, separators=(",", ":")))Configure your logger to not add its own formatting. If you use structlog, set renderer=structlog.processors.JSONRenderer. In TypeScript:
console.log(JSON.stringify({ ...record, schema_version: 2 }));9. Design for the queries you will run
Your schema should answer the three questions every on-call engineer asks: what model, how much, how long, and did it fail? Write the schema backward from the query.
SELECT served_model, percentile(duration_ms, 0.95) as p95
FROM llm_logs
WHERE env = 'prod' AND started_at > now() - interval 1 hour
GROUP BY served_model;Or with jq:
jq 'select(.error != null) | .provider' llm.ndjson | sort | uniq -cIf you cannot express the query without nested loops in your log UI, flatten the field.
10. Common pitfalls and tradeoffs
Over-nesting. A depth of more than three levels slows querying in most log stores. Keep tokens, error, and timing as the only objects.
Logging raw streaming chunks. If you stream, log the final aggregated result, not each token. Per-token logs are only useful in a dedicated trace sampler with sampling rates below 1%.
Ignoring cache-control hints. Providers may return cache_creation or cache_read metrics. If you forward cache-control hints from the client, log whether the provider honored them. That’s a first-class field in your JSON logging schema for LLM requests.
Mixing sync and async IDs. In async flows, generate the request_id at submission, not at completion. Otherwise you can’t correlate timeouts.
Forgetting schema versioning. Add a schema_version integer. When you add cached_prompt, bump it. Downstream dashboards break silently without this.
{
"schema_version": 2,
"request_id": "req_01h9x8n3k2"
}Storing cost instead of usage. Hard-coding price in the log forces rewrites when contracts change. Store raw counts; compute cost in the warehouse.
No sampling strategy. At high QPS, full logging of every request may cost more than the LLM calls. Sample 10% of successful stop reasons, but log 100% of errors and length truncations.
11. Minimal complete example
{
"schema_version": 2,
"request_id": "req_01h9x8n3k2",
"trace_id": "trace_88a",
"started_at": "2024-05-12T18:22:01.123Z",
"ended_at": "2024-05-12T18:22:04.987Z",
"duration_ms": 3864,
"requested_model": "gpt-4o",
"served_model": "gpt-4o-2024-05-13",
"provider": "openai",
"routing_directive": "prefer-cache",
"tokens": {
"prompt": 412,
"completion": 188,
"total": 600,
"cached_prompt": 256
},
"finish_reason": "stop",
"truncated": false,
"prompt_ref": "s3://llm-logs/req_01h9x8n3k2/prompt.json",
"response_ref": "s3://llm-logs/req_01h9x8n3k2/response.json",
"app": "support-bot",
"env": "prod",
"user_tier": "pro"
}Adopt this JSON logging schema for LLM requests as a baseline, then extend per your compliance needs. The goal is a log line that answers “what model, how much, how long, and did it fail” without a JOIN.