Logging LLM outputs to catch quality regressions early

Shipping a prompt change without logging llm outputs quality regression is like deploying code without tests. A model swap or provider update can silently degrade answers, and you’ll only hear about it from angry users. This guide gives concrete steps to capture outputs, build a baseline, and catch drift before it hits production.

Step 1: Define the regression surface

You can’t catch a quality regression if you don’t know what “good” looks like. Start by enumerating the fields that matter for your use case. At minimum, log the request payload, the model identifier, the raw completion, finish reason, token usage, and latency. If you serve multiple tenants, add a tenant_id and a prompt_version tag so you can slice regressions by cohort.

A flat JSON schema works for early stages:

{
  "ts": 1715270000.123,
  "call_id": "a1b2c3d4",
  "model": "gpt-4o-mini",
  "prompt_version": "v12",
  "messages": [{"role": "user", "content": "Summarize: ..."}],
  "completion": "The article discusses...",
  "finish_reason": "stop",
  "usage": {"prompt_tokens": 120, "completion_tokens": 45, "total_tokens": 165},
  "latency_ms": 820
}

Effective logging llm outputs quality regression means capturing metadata that explains why an output changed, not just the text. Add seed and temperature if you rely on deterministic sampling. If you use cached prompts, record cached_tokens so you can separate cost regressions from quality regressions.

Step 2: Instrument the call path

Wrap your LLM client so every completion flows through one logging function. Avoid ad-hoc prints scattered across handlers. Below is a minimal Python wrapper around the OpenAI SDK. When you route through n4n.ai, the OpenAI-compatible endpoint returns standard usage objects and honors client routing directives, so the same logging code works across 240+ models without branching.

import json, time, hashlib, openai

client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

def logged_chat(messages, model, prompt_version="v0", **kwargs):
    start = time.time()
    resp = client.chat.completions.create(model=model, messages=messages, **kwargs)
    elapsed = (time.time() - start) * 1000
    rec = {
        "ts": start,
        "call_id": hashlib.sha256(f"{messages}{start}".encode()).hexdigest()[:16],
        "model": model,
        "prompt_version": prompt_version,
        "messages": messages,
        "completion": resp.choices[0].message.content,
        "finish_reason": resp.choices[0].finish_reason,
        "usage": resp.usage.model_dump(),
        "latency_ms": elapsed,
    }
    with open("llm_logs.jsonl", "a") as f:
        f.write(json.dumps(rec) + "\n")
    return resp

Call logged_chat everywhere instead of client.chat.completions.create. The wrapper adds negligible overhead (sub-millisecond JSON serialization) and gives you a single tail-able file: tail -f llm_logs.jsonl | jq '.finish_reason'.

Step 3: Persist to a queryable store

JSONL is fine for a laptop, but regression analysis needs SQL. Load the file into DuckDB or Postgres nightly, or stream via a worker. Schema:

CREATE TABLE llm_calls (
  ts             TIMESTAMP,
  call_id        TEXT PRIMARY KEY,
  model          TEXT,
  prompt_version TEXT,
  completion     TEXT,
  finish_reason  TEXT,
  prompt_tokens  INT,
  completion_tokens INT,
  latency_ms     DOUBLE
);

With data in a table, you can answer “did v13 of the summarizer produce shorter completions on average?” in one query:

SELECT prompt_version, AVG(completion_tokens) AS avg_out
FROM llm_calls
WHERE model = 'gpt-4o-mini'
GROUP BY prompt_version ORDER BY prompt_version;

Logging llm outputs quality regression becomes tractable when you can join completions to deploy timestamps from your CI system.

Step 4: Establish a baseline and measure drift

Pick a set of “golden” prompts that represent real traffic. Store their known-good completions as baseline. On every deploy, replay those prompts in a staging environment and compare.

Text similarity is a cheap first signal. Use difflib to get a ratio without extra model calls:

import difflib

def similarity(a: str, b: str) -> float:
    return difflib.SequenceMatcher(None, a, b).ratio()

baseline = "The cat sat on the mat."
candidate = "A cat was sitting on a mat."
print(similarity(baseline, candidate))  # ~0.82

Set a threshold (e.g., 0.85) per prompt. If a new build drops below it, flag for human review. For hallucination-specific checks, run a lightweight classifier or a second LLM judge on the candidate and log its score alongside.

If you have embeddings, cosine distance is more robust to paraphrasing:

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Store the baseline embedding once; compute candidate embeddings in the regression job.

Step 5: Run scheduled regression checks

A one-off script isn’t enough. Regressions also come from provider-side model updates you didn’t trigger. Schedule a cron job that runs the golden set every 6 hours against production routing.

# crontab
0 */6 * * * /usr/bin/python3 /opt/llm_regression/run_checks.py >> /var/log/llm_regression.log 2>&1

Inside run_checks.py, load the latest prompt_version from your deploy API, replay golden prompts via logged_chat, compute similarity, and emit a metrics line:

import sys, json, time
from your_module import logged_chat, similarity, BASELINE

def main():
    worst = 1.0
    for item in BASELINE:
        resp = logged_chat(item["messages"], item["model"], prompt_version="prod")
        comp = resp.choices[0].message.content
        s = similarity(item["expected"], comp)
        worst = min(worst, s)
    print(f"regression_min_similarity={worst:.3f}")
    if worst < 0.85:
        sys.exit(1)  # alerting picks this up

if __name__ == "__main__":
    main()

Pipe the exit code to your existing alerting (PagerDuty, Slack, Grafana). The key is that logging llm outputs quality regression is continuous, not a pre-deploy gate only.

Step 6: Alert and tie to deploys

When similarity drops, you need context fast. Enrich the alert with the call_id and the prompt_version active at that timestamp. Query your llm_calls table:

SELECT call_id, model, prompt_version, completion
FROM llm_calls
WHERE ts > NOW() - INTERVAL '6 hours'
  AND finish_reason != 'stop'
ORDER BY ts DESC LIMIT 5;

Non-stop finish reasons often precede truncated or degraded answers. If the regression correlates with a prompt_version change, roll back that prompt. If it correlates with a model string change but no deploy on your side, the provider shipped a silent update—your log just gave you proof.

Verify success

You’ve built the pipeline; now prove it fires. Two checks:

1. Inject a bad model. Point logged_chat at a deliberately weak model (e.g., a tiny instruct model for a reasoning task) in staging. The similarity score should crater and the cron job should exit non-zero within one cycle.
2. Replay a known drift. Take a baseline completion, manually edit it to be semantically wrong, and place it in the candidate slot of your checker. The script should report < threshold and emit the alert.

If both tests produce alerts with call_ids you can trace in your store, your logging llm outputs quality regression system is operational. From here, extend the golden set monthly as new user intents appear, and keep the similarity thresholds reviewed by a human quarterly—automated signals reduce noise, they don’t eliminate judgment.