Every enterprise that routes prompts through a shared intermediary needs an audit log LLM API gateway that records who called what, when, and with which outcome. Skip this and you fail SOC 2, GDPR subject access requests, and basic incident response. This guide lays out an ordered path to implement and verify that logging without drowning in noise.
1. Catalog the events that actually matter
Start by listing the state transitions you must capture. A request enters the gateway, gets authenticated, resolves to a model and provider, possibly falls back, returns a response or error, and consumes tokens. Each of those steps is an auditable event.
Most teams log only the happy path. That is a mistake. You need:
- Authentication success and failure (including rejected API keys)
- Request accepted with model and routing directive
- Provider attempt, retry, and fallback
- Response status (200, 429, 500, timeout)
- Token metering (input, output, cached)
- Cache hit/miss against provider or gateway cache
If your gateway exposes an OpenAI-compatible endpoint, require the client to send an X-Request-Id header and echo it back. Without a stable correlation id owned by the caller, you fragment traces across retries and fallback and cannot answer “what happened to the request my service sent at 08:23:01.123?”
A minimal structured record looks like this:
{
"ts": "2024-05-12T08:23:01.123Z",
"req_id": "req_8f2c",
"actor": {"api_key_id": "key_123", "tenant": "acme"},
"action": "completion",
"model_requested": "gpt-4o",
"model_resolved": "gpt-4o-2024-05-13",
"provider": "openai",
"provider_attempts": ["openai", "azure-openai"],
"status": 200,
"tokens_in": 120,
"tokens_out": 45,
"cache_hit": false,
"latency_ms": 820
}Store this as one line per event, not one document per session. Session reconstruction comes from the req_id.
2. Decide what payload data to retain
Logging full prompts and completions is the fastest way to create a compliance liability. PII, secrets, and copyrighted text end up in your log sink. Conversely, logging nothing about content makes abuse investigation impossible.
The pragmatic middle ground: store a length, a hash, and redacted spans. Keep the hash so you can prove two requests carried identical prompts without exposing the text.
import hashlib
def log_safe_prompt(prompt: str) -> dict:
return {
"prompt_sha256": hashlib.sha256(prompt.encode()).hexdigest(),
"prompt_len": len(prompt),
"redacted_spans": count_cardinal_numbers(prompt) # your own redactor
}Redaction vs hashing
Hashing proves equality but reveals nothing about content. Redaction keeps some context for analysts but expands the surface for leakage. Run redaction as a deterministic pipeline step before the log writer; never log the original object reference alongside the redacted copy.
If regulations require content retention (some finance use cases do), isolate that store behind extra encryption and shorter retention. Do not mix it with the operational audit log LLM API gateway stream.
3. Make the log append-only and verifiable
An audit log you can edit is not an audit log. Write to a sink that forbids mutation: object-locked S3, WriteOnce Elasticsearch, or a dedicated ledger service. Kafka is not immutable by default—its log can be compacted or deleted—so do not treat a raw topic as your system of record.
For self-hosted setups, hash-chain entries so each line commits to the previous.
Example S3 object lock:
aws s3api put-object --bucket audit-logs --key 2024-05-12/req_8f2c.json \
--body req_8f2c.json --object-lock-mode COMPLIANCEAdd a signature per batch:
import hmac, hashlib, json
def sign_batch(lines: list[str], secret: bytes) -> str:
h = hmac.new(secret, msg=json.dumps(lines).encode(), digestmod="sha256")
return h.hexdigest()Ship the signature out-of-band (separate bucket or KMS). If an attacker tampers with a line, the chain breaks on the next entry.
4. Track multi-provider routing and fallback
Modern gateways sit in front of many model providers. A request for gpt-4o might start at OpenAI, hit a 429, and finish on Azure OpenAI. Your audit log LLM API gateway must record the full attempt list and the final winner.
A gateway such as n4n.ai that performs automatic fallback when a provider is rate-limited must emit both the intended and final route in the same audit record; otherwise you cannot explain why a request billed to one model executed on another. The same applies when a client sends an explicit routing directive (x-router-prefer: anthropic) that the gateway honors before falling back.
Capture provider_attempts and provider_selected as distinct fields. If you only log the winner, post-incident analysis of a bad provider release becomes blind. Also record the client’s routing directive verbatim—this is the only way to prove the gateway obeyed its contract.
5. Enforce retention and least-privilege access
Decide retention up front. Security teams often want seven years; storage costs and data-minimization laws push back. A common pattern: hot log for 90 days in Elasticsearch for detection, cold immutable copy for the regulated window.
Access control is separate from the application DB. The on-call engineer should not have delete rights on audit buckets. Use IAM policies that allow s3:GetObject but not s3:DeleteObject for human roles; only the compliance service principal can transition objects to expire.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": "arn:aws:s3:::audit-logs/*"
}]
}Pitfall: granting the log pipeline role broad write and then reusing it for replay jobs. Split the writer and the reader. GDPR “right to erasure” rarely applies to audit logs kept for legal obligation, but you need that exemption documented in your retention policy.
6. Test the log under failure conditions
Most logging code paths are happy-path tested and never exercised when the gateway itself is degraded. Write adversarial tests:
- Kill the upstream provider and assert a fallback event with
status: 503from gateway is logged. - Send an expired API key and confirm an auth failure event, not silent drop.
- Inject clock skew of ±2 minutes and verify timestamps stay monotonic per
req_id. - Truncate the request body and ensure the log still captures
actorandaction. - Simulate object-lock failure on the sink and confirm the gateway fails open with a local dead-letter, not silent loss.
A missing negative event is how breaches go unnoticed. If the gateway returns 401, the audit log LLM API gateway must contain that 401 with the rejected key id.
7. Connect logs to detection and billing
Audit is not just for auditors. Per-token metering in the log feeds both cost allocation and anomaly detection. Gateways that forward provider cache-control hints—n4n.ai does this—emit a cache_hit flag; reconcile that against provider invoices to catch metering drift. Cached tokens are often priced differently, so a mismatch is either a billing bug or a logging gap.
Query for a tenant whose tokens_out spikes 10x versus weekly baseline:
SELECT tenant, sum(tokens_out) AS toks
FROM audit_logs
WHERE ts > now() - interval '1 day'
GROUP BY tenant
HAVING sum(tokens_out) > 1000000;Forward the same stream to your SIEM with a detection rule on provider_attempts length > 3 (indicates persistent degradation). Because the gateway honors client routing directives, a sudden flood of x-router-prefer overrides may signal a tenant gaming fallback logic.
Common tradeoffs to accept
High-cardinality fields like full prompt text give investigation power but blow up storage and risk PII leaks. Hash and length are lossy but safe. Immutable storage costs more than a Postgres table; treat it as non-negotiable for enterprise.
Latency overhead from signing each request is real. Batch sign every 100 ms instead of per call, and measure the p99 impact. A few hundred microseconds is acceptable for the compliance guarantee.
An audit log LLM API gateway is only useful if it survives the incidents it is meant to explain. Build the pipeline first, test it broken, then ship the features.