What to log for chat completion calls is the set of structured request, response, and infrastructure metadata you record for every invocation of an OpenAI-compatible /v1/chat/completions endpoint. At minimum, this means capturing the model string, token counts, latency, finish reason, and any routing or cache hints—so you can debug failures, attribute cost, and evaluate output quality after the fact.
Definition: what to log for chat completion calls
The phrase “what to log for chat completion calls” describes the explicit schema of observability data for LLM chat API traffic. Unlike traditional HTTP logging that stops at status code and path, LLM calls require semantic fields: the exact messages sent, the completion returned, token accounting, and the provider routing path.
If you treat an OpenAI-compatible call like a black-box POST, you lose the ability to reproduce a bad response or explain a usage spike. Structured logging turns each call into a queryable event that survives provider dashboard changes.
How OpenAI-compatible logging works
An OpenAI-compatible chat completion call accepts a JSON body with model, messages, temperature, and other sampling params. The response returns id, model, choices, usage, and sometimes system_fingerprint. The logging layer sits between your code and the HTTP client, capturing both sides plus timing.
// Request
{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Summarize: ..."}],
"temperature": 0.2,
"max_tokens": 200
}// Response (non-streaming)
{
"id": "chatcmpl-123",
"model": "gpt-4o-mini",
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": "..."},
"finish_reason": "stop"
}],
"usage": {"prompt_tokens": 12, "completion_tokens": 34, "total_tokens": 46}
}For streaming responses, the usage often arrives in the final data: [DONE] event or a trailing usage chunk. Your logger must reassemble the stream to capture token counts.
Error responses
A 429 or 500 still costs you latency. Log the status, error_code, and retry_after header. If the call failed before response parse, log the exception type and the request payload size.
Core fields to capture
Request fields
model: the requested model string.messages: the full conversation array (or a hash if privacy-bound).temperature,top_p,max_tokens: sampling config.request_id: your own correlation id, ideally propagated from upstream.
Response fields
response_id: the provider’s id.returned_model: the model that actually served the call (may differ under fallback).finish_reason:stop,length,content_filter,tool_calls, etc.usage.prompt_tokens,usage.completion_tokens: for cost.latency_ms: measured client-side from send to final byte.
Infrastructure & routing metadata
upstream_provider: e.g., openai, anthropic, or a gateway.cache_hit: whether provider cache-control was honored.routing_directive: any client-specified region or provider pin.system_fingerprint: provider-specific build marker.
If you use a gateway such as n4n.ai, the response includes the fulfilled provider and per-token metering; log those fields to attribute spend correctly when automatic fallback shifts the call to a different backend. The gateway also forwards cache-control hints, so cache_hit reflects reality, not assumption.
Why structured logging matters for LLM APIs
LLM failures are rarely 500 errors. They are silent quality regressions: a prompt tweak drops accuracy, a fallback model changes tone, or cached tokens quietly expire. Without per-call logs you cannot isolate which change caused the drift.
Second, token accounting is the only way to control cost. Provider dashboards lag and aggregate by account, not by your tenant. Your own logs give per-request granularity aligned with product dimensions.
Third, evaluations need raw data. You cannot run an offline benchmark on last week’s traffic if you didn’t store the messages and completions. Structured logs are the sampling frame for eval sets.
Fourth, incident response shrinks from days to minutes. When a customer reports “the bot sounded weird Tuesday,” you query returned_model and finish_reason for that tenant and time range.
Concrete example: logging middleware
Below is a minimal Python wrapper using httpx and structlog. It logs the essential schema without blocking the response.
import time, json, structlog
log = structlog.get_logger()
async def logged_chat(client, payload, tenant_id):
start = time.monotonic()
try:
resp = await client.post("/v1/chat/completions", json=payload)
elapsed = (time.monotonic() - start) * 1000
body = resp.json()
log.info("chat_completion",
tenant_id=tenant_id,
requested_model=payload.get("model"),
returned_model=body.get("model"),
finish_reason=body["choices"][0]["finish_reason"],
prompt_tokens=body["usage"]["prompt_tokens"],
completion_tokens=body["usage"]["completion_tokens"],
latency_ms=round(elapsed, 1),
response_id=body.get("id"),
status=resp.status_code,
upstream_provider=resp.headers.get("x-upstream-provider"),
cache_hit=resp.headers.get("x-cache-hit"),
)
return body
except Exception as e:
log.error("chat_completion_failed",
tenant_id=tenant_id,
requested_model=payload.get("model"),
error=str(e),
latency_ms=round((time.monotonic()-start)*1000,1),
)
raiseA single emitted event looks like:
{
"event": "chat_completion",
"tenant_id": "acme",
"requested_model": "gpt-4o-mini",
"returned_model": "gpt-4o-mini",
"finish_reason": "stop",
"prompt_tokens": 12,
"completion_tokens": 34,
"latency_ms": 412.3,
"response_id": "chatcmpl-123",
"status": 200,
"upstream_provider": "openai",
"cache_hit": "true"
}Add messages and completion only if your compliance policy allows. Store them in an object store keyed by response_id rather than in the hot log stream.
Streaming and partial responses
Streaming complicates what to log for chat completion calls. The finish reason and usage appear at the end. Buffer the stream, count tokens from the final usage chunk, and log once when the stream closes. If the connection drops mid-stream, log finish_reason: "stream_error" and the bytes received.
Common misconceptions
“Log only the status code.” A 200 with finish_reason: length means the model truncated. You paid for tokens but got a partial answer. Status alone hides this.
“The model field in the request is what served the response.” Under rate limits or degraded providers, a gateway may fulfill with a fallback model. The response model field (or gateway metadata) is truth. Log both requested and returned.
“Token counts are exact across providers.” Tokenization differs. gpt-4o-mini and claude-3-haiku count the same string differently. Log the provider alongside usage so cost math stays correct.
“Full prompt logging is always unsafe.” It depends on the data class. For internal tooling, storing prompts is the only way to replay eval. For PII, hash or redact, but keep a stable content hash to detect prompt changes.
“Cache hits don’t need logging.” Provider cache-control hints can cut cost dramatically. If you don’t log cache_hit and the cache key, you can’t tell whether a prompt refactor broke caching.
“Logging slows the hot path.” A single JSON emit per call is microseconds. The cost is in storing full message bodies; keep those out of the synchronous log.
Retention and sampling
You do not need every prompt forever. Use tiered retention: hot logs with metadata for 30 days, full payloads in cheap storage for 90. Sample 10% of successful calls if volume is extreme, but always keep errors and finish_reason != "stop".
A practical logging checklist
Before shipping to production, confirm each chat completion call emits:
- Correlation id (tenant + request)
- Requested vs returned model
- Full token usage (prompt, completion, total)
- Finish reason and latency
- Upstream provider and any routing directive
- Cache hit flag
- Optional: message hash or stored payload reference
What to log for chat completion calls is not a nice-to-have; it is the difference between operating an LLM feature blind and iterating on evidence. Start with the schema above, keep the hot path thin, and archive full payloads separately.
Wrapping up
Treat logs as the source of truth for every OpenAI-compatible interaction. The definition is simple: capture the semantic fields that describe the call, the response, and the route it took. The engineering payoff is debugging, cost, and quality control in one pipeline.