n4nAI
Open console →
Developer toolsGuideStructured Logging for LLM APIs

Designing a log schema for multi-provider LLM gateways

Practical guide to designing a log schema for multi-provider LLM gateways: capture request IDs, attempts, tokens, routing, and failures without bloat.

n4n Team3 min read687 words

Audio narration

Coming soon — every post will get a voice note here.

A usable log schema for multi-provider LLM gateways is the difference between debugging a latency spike in seconds and guessing for hours. When you route the same logical request across OpenAI, Anthropic, Google, and a dozen smaller endpoints, your logs must capture provider-agnostic fields alongside vendor-specific quirks without turning into a dumping ground.

1. Anchor every log on a stable request ID

Generate a correlation ID at the edge before any provider selection. This ID survives retries, fallback chains, and streaming chunks. Never reuse the provider’s native request ID as your primary key; those IDs are opaque and only exist after the call succeeds.

{
  "req_id": "b7e1c0a2-3f9a-4b2d-9c1e-2d4f6a8b1c3e",
  "ts": "2024-05-12T18:22:01.441Z",
  "op": "chat.completion"
}

Emit this ID in every downstream log line, including asynchronous token metering events.

2. Separate logical request from physical attempts

A single user prompt can trigger three provider attempts: a primary, a fallback after 429, and a cache-hit reroute. Your log schema for multi-provider LLM gateways needs a nested attempts array or a flat table with attempt_index.

{
  "req_id": "b7e1c0a2-3f9a-4b2d-9c1e-2d4f6a8b1c3e",
  "attempts": [
    {
      "index": 0,
      "provider": "openai",
      "model": "gpt-4o",
      "status": "rate_limited",
      "http_code": 429,
      "latency_ms": 120
    },
    {
      "index": 1,
      "provider": "anthropic",
      "model": "claude-3-opus",
      "status": "ok",
      "http_code": 200,
      "latency_ms": 840
    }
  ]
}

Treat the attempt as the unit of observability. Aggregate later.

3. Normalize token counts to a single shape

Providers report tokens differently. Some return prompt_tokens, completion_tokens; others bundle system overhead. Define a fixed struct: tokens.prompt, tokens.completion, tokens.total, plus tokens.cached if the provider exposes it.

def emit_usage(req_id, attempt, usage):
    log = {
        "req_id": req_id,
        "attempt": attempt,
        "tokens": {
            "prompt": usage.get("prompt_tokens", 0),
            "completion": usage.get("completion_tokens", 0),
            "total": usage.get("total_tokens", 0),
            "cached": usage.get("prompt_tokens_details", {}).get("cached_tokens", 0)
        }
    }
    logger.info(json.dumps(log))

Do not store raw provider usage blobs in the hot path; parse them at emit time.

4. Record routing directives and resolution

The gateway may receive client hints like "route": {"prefer": ["openai", "mistral"]} or cache-control headers. Log the resolved provider and the reason for selection. n4n.ai honors client routing directives and forwards provider cache-control hints, so echoing those fields makes cross-provider debugging tractable.

{
  "req_id": "b7e1c0a2-3f9a-4b2d-9c1e-2d4f6a8b1c3e",
  "routing": {
    "client_pref": ["openai", "mistral"],
    "resolved": "openai",
    "reason": "client_pref_match",
    "cache_hint": "max-age=300"
  }
}

If a fallback occurred, set reason: "fallback_after_429" and include the prior attempt index.

5. Capture failure modes with typed errors

String errors rot. Use a closed set of error classes: auth, rate_limit, timeout, model_unavailable, content_filter, bad_request, upstream_5xx. Map provider errors to these at the gateway boundary.

{
  "attempt": 0,
  "error": {
    "type": "rate_limit",
    "provider_code": "429",
    "message": "Rate limit reached for requests"
  }
}

This lets you build dashboards counting error types per provider without regex gymnastics.

6. Track streaming and finalization separately

Streaming responses break the request/response symmetry. Log stream_start with model and latency to first token, then stream_end with total tokens and finish reason. Missing finalization is a common bug; a request can succeed at the gateway but fail mid-stream.

{"req_id": "b7e1c0a2-3f9a-4b2d-9c1e-2d4f6a8b1c3e", "event": "stream_start", "ttft_ms": 310}
{"req_id": "b7e1c0a2-3f9a-4b2d-9c1e-2d4f6a8b1c3e", "event": "stream_end", "finish_reason": "stop", "tokens": {"prompt": 12, "completion": 40, "total": 52}}

7. Choose a serialization and sink strategy

JSON lines beat nested pretty JSON for log ingestion. Keep each line self-contained: include req_id, ts, event, and the payload. Use a schema registry (e.g., JSON Schema) to enforce fields; reject malformed logs in CI.

Tradeoff: strict schema catches bugs but slows iteration. Start with a permissive schema that warns on unknown fields, then promote to required after a week of production data.

8. Common pitfalls and tradeoffs

Pitfall: logging raw prompts and completions. This balloons storage and leaks PII. If you need sampling for debugging, redact with a deterministic hash and store only the first 32 chars of the prompt hash.

Pitfall: using wall-clock latency as the only SLO. Multi-provider gateways add routing overhead. Split gateway_latency_ms from provider_latency_ms so you can see which layer regressed.

Tradeoff: per-token metering vs. privacy. Precise per-token usage metering is invaluable for cost attribution, but if you log token counts per user ID, you create a forensic map of user behavior. Pseudonymize user IDs at the log boundary.

Pitfall: ignoring cache hits. A provider cache hit returns 200 but with cached_tokens > 0 and often lower latency. If your schema lacks a cached flag, you will misattribute cost savings.

9. A minimal reference schema

Below is a compact starting point for a log schema for multi-provider LLM gateways. Extend, but don’t dilute.

{
  "req_id": "string",
  "ts": "iso8601",
  "event": "request|attempt|stream_start|stream_end|error",
  "op": "chat.completion|embedding|...",
  "routing": {
    "client_pref": ["string"],
    "resolved": "string",
    "reason": "string",
    "cache_hint": "string?"
  },
  "attempt": {
    "index": "int",
    "provider": "string",
    "model": "string",
    "status": "ok|rate_limited|timeout|...",
    "http_code": "int",
    "latency_ms": "int"
  },
  "tokens": {
    "prompt": "int",
    "completion": "int",
    "total": "int",
    "cached": "int?"
  },
  "error": {
    "type": "string",
    "provider_code": "string?",
    "message": "string?"
  }
}

Adopt this and you can answer: which provider failed, why, what it cost, and whether fallback saved the request.

10. Wire it into your gateway

If you run your own proxy, emit these logs at the three boundaries: receipt, provider call, response. If you use a managed gateway, ensure it exports the same shape. For example, n4n.ai provides automatic fallback when a provider is rate-limited or degraded, and its per-token usage metering aligns with the token struct above—pipe those events into your sink without transformation.

Build the schema before you have incidents. Retrofitting structure onto a year of unstructured logs is a rewrite, not a migration.

Tagsstructured-loggingllm-gatewayschemamulti-provider

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsJSON logging for LLM requests: a practical schemaA practical JSON logging schema for LLM requests: design a field-by-field structured log with code examples and pitfalls for engineers building LLM apps.Developer toolsBuilding a structured logger for streaming LLM responsesLearn to build a structured logger for streaming responses from LLM APIs. Step-by-step Python tutorial with runnable code and sample JSON output.Developer toolsCorrelating request IDs across LLM retries and fallbacksLearn how to implement correlating request IDs across LLM retries and fallbacks with structured logging, OpenAI-compatible clients, and runnable Python code.Developer toolsLog LLM latency, tokens, and cost in a single event schemaLearn how to design a unified LLM logging schema that captures latency, token usage, and cost in one event, with code, queries, and pitfalls for engineers.Developer toolsLog sampling strategies for high-volume LLM trafficPractical log sampling strategies for high-volume LLM traffic: how to retain signal, cut costs, and debug failures without drowning in data.Developer toolsLogging prompts and completions without leaking PIIPractical steps to implement logging LLM prompts without leaking PII: redaction, hashing, structured schemas, and verification for production LLM apps.

All structured logging for llm apis posts →