Most teams wire up LLM API calls and only later realize their request and response logs are full of API keys, email addresses, and customer PII. Redacting sensitive data in LLM logs is not optional if you handle real user input or regulated data; a single unredacted completion can become a compliance incident. This guide walks through a concrete pipeline you can drop into your service today.
Step 1: Capture logs at the boundary
Centralize logging in one middleware layer instead of scattering print statements through your call sites. At the HTTP boundary you see the exact payloads sent to and received from the model, which is where leaks happen. Deep-stack logging misses headers, proxy additions, and transformed messages.
In a FastAPI app, write a middleware that clones the request body and response body, runs them through a redactor, then emits a structured log line. Keep the redaction synchronous and deterministic so the log sink never receives raw data.
import json
from starlette.middleware.base import BaseHTTPMiddleware
class RedactingLogger(BaseHTTPMiddleware):
async def dispatch(self, request, call_next):
body = await request.body()
response = await call_next(request)
resp_body = b""
async for chunk in response.body_iterator:
resp_body += chunk
redacted_req = redact(json.loads(body) if body else {})
redacted_resp = redact(json.loads(resp_body) if resp_body else {})
logger.info("llm_proxy", extra={"req": redacted_req, "resp": redacted_resp})
# Rebuild response stream for the client
response.body_iterator = iter([resp_body])
return responseThe key point: redact before the logger.info call, never after. If you log first and redact later in a separate processor, a crash or misconfig leaves raw data on disk.
Step 2: Define a redaction contract
You cannot redact what you have not named. Start with three buckets:
- Credentials:
api_key,authorization,bearertokens, internal signing secrets. - PII: email, phone, SSN, free-text customer input that may contain any of those.
- Business secrets: internal account IDs, prompt templates with embedded keys, unreleased model names.
Structure your logs as explicit fields so the redactor can target paths instead of guessing. A flat message string is unredeemable; a nested object is trivial.
{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "my email is a@b.com"}],
"metadata": {"api_key": "sk-1234", "trace_id": "abc"}
}Decide up front: trace_id is safe, api_key is never. Encode that policy in code, not in a wiki.
Step 3: Build a deterministic redactor
A regex-based pass is enough for 80% of cases. Extend with a deny-list of JSON keys that must always be masked, and recursively walk the structure so nested dictionaries and lists are covered.
import re
EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
KEY_RE = re.compile(r"sk-[A-Za-z0-9]{20,}")
DENY_KEYS = {"api_key", "authorization", "password", "token"}
def redact(obj):
if isinstance(obj, dict):
return {
k: ("***REDACTED***" if k.lower() in DENY_KEYS else redact(v))
for k, v in obj.items()
}
if isinstance(obj, list):
return [redact(v) for v in obj]
if isinstance(obj, str):
return KEY_RE.sub("***REDACTED***", EMAIL_RE.sub("***REDACTED***", obj))
return objThis function is pure and unit-testable. Run it on every log record. For free-text PII beyond emails and keys, add a lightweight NER pass with a library like presidio-analyzer if your compliance scope demands it, but start with the regex and deny-list to ship fast.
Step 4: Wrap your LLM client
If you call an OpenAI-compatible endpoint directly from Python, subclass the client or use a wrapper that redacts before logging. When you route through a gateway like n4n.ai, you get per-token usage metering and automatic fallback across 240+ models, but the gateway does not mask prompt content—redacting sensitive data in LLM logs stays your responsibility at the application layer. The wrapper pattern works identically regardless of backend.
from openai import OpenAI
class RedactingClient:
def __init__(self, base_url, api_key):
self.client = OpenAI(base_url=base_url, api_key=api_key)
def create_chat(self, **kwargs):
log_payload = redact(kwargs)
logger.info("llm_request", extra=log_payload)
resp = self.client.chat.completions.create(**kwargs)
logger.info("llm_response", extra=redact(resp.model_dump()))
return respSwap the raw OpenAI instance for RedactingClient in your dependency injection. For async code, mirror the method with await and resp.model_dump() on the async response object.
Step 5: Handle streaming responses
Streaming breaks the simple “log the whole body” approach. You must either buffer the full stream and redact at the end, or redact each chunk (riskier for cross-chunk PII). Buffering is safer for compliance and usually acceptable because you are already holding the text in memory to yield to the caller.
async def stream_redacted(client, **kwargs):
chunks = []
stream = await client.chat.completions.create(stream=True, **kwargs)
async for chunk in stream:
chunks.append(chunk)
full = "".join(c.choices[0].delta.content or "" for c in chunks)
logger.info("llm_stream", extra={"content": redact(full)})
return chunksIf you cannot buffer due to latency, redact per chunk but accept that a split email may leak half. Not worth it for most teams. Prefer buffering and log the redacted full text after the stream closes.
Step 6: Write verification tests
Redaction code rots. Lock it with a test that fails if a known secret reaches the log sink. Use caplog to capture handler output and assert on absence of raw strings.
def test_redact_emails(caplog):
import logging
with caplog.at_level(logging.INFO):
logger.info("test", extra=redact({"msg": "mail me at x@y.com"}))
assert "x@y.com" not in caplog.text
assert "***REDACTED***" in caplog.text
def test_redact_api_key(caplog):
with caplog.at_level(logging.INFO):
logger.info("test", extra=redact({"api_key": "sk-abcdefghijklmnopqrstuvwx"}))
assert "sk-abcdefghijklmnopqrstuvwx" not in caplog.textRun these in CI on every commit. Add a property-based test with hypothesis that generates random strings containing emails and keys to catch regex drift. Redacting sensitive data in LLM logs is only trustworthy if the tests are mandatory.
Step 7: Audit and monitor redaction coverage
Log a counter of how many redactions occurred per request. If the count suddenly drops to zero on a path that normally contains PII, your redactor may be bypassed by a new field name or a refactored client.
def redact_with_count(obj):
before = json.dumps(obj)
after = redact(obj)
redacted = before.count("***REDACTED***")
logger.info("redaction_stats", extra={"count": redacted})
return afterWire this into your metrics pipeline. Alert on zero redactions for user-facing endpoints over a rolling window. Sample raw-vs-redacted diffs in a secure debug bucket if you need to tune the regex, but never store the raw sample long-term.
Verify success
After deploying, do three concrete things:
- Send a test request containing
sk-test1234567890abcdefanduser@domain.comthrough your endpoint. - Inspect the emitted log line in your local file or log UI. Confirm both values appear as
***REDACTED***and the raw strings are absent. - Run
grep -R "sk-" ./logsacross a day of production logs. Any hit means a bypass; fix the path and add a regression test.
Redaction is a continuous control, not a one-time patch. Treat the redactor as production code: versioned, tested, and monitored. The moment you stop verifying, the logs quietly fill with the data you promised your customers you would protect.