How to triage degraded LLM response quality incidents

Triaging degraded LLM response quality incidents requires a different playbook than a hard 500 error. When the API returns 200 but the JSON is malformed, the tone is wrong, or the model quietly drops constraints, you are debugging a moving target across providers, model versions, and prompt changes. The first rule is to measure before you touch anything.

Step 1: Confirm the degradation is real and quantify it

Do not trust a single angry user report or a spike in support tickets. Pull your completion logs for the last 24–72 hours and compute the failure rate against a deterministic check that matches your application’s contract. If you expect structured output, parse it. If you expect a sentiment label, validate the enum.

import json
from collections import Counter

def contract_ok(raw: str) -> bool:
    try:
        obj = json.loads(raw)
        return "city" in obj and "population" in obj
    except ValueError:
        return False

logs = [json.loads(l) for l in open("completions.log")]
counter = Counter(contract_ok(l["response"]) for l in logs)
fail_rate = 1 - counter[True] / len(logs)
print(f"total={len(logs)} fail_rate={fail_rate:.3f}")

If your contract failure rate jumped from 0.2% to 4%, you have an incident. Define the baseline from the previous week’s same weekday, not just yesterday, because volume swings distort ratios.

Verify success

You have a numeric delta, a start timestamp from log analysis, and a reproducible metric. If the rate is flat, the problem is upstream of the model (UI, retrieval, auth) and you should stop reading this triage guide.

Step 2: Isolate prompt, model, or provider

A degradation can originate from three layers: your prompt template changed, the model weights or decoding were updated, or the provider routed you to a degraded replica. Run a golden set of 50 inputs that previously passed your eval through the exact same prompt and a pinned model version.

Capture the system_fingerprint field if your provider returns it. A changed fingerprint means the model backend changed without notice.

from openai import OpenAI
client = OpenAI()  # base_url defaults to api.openai.com or your gateway

resp = client.chat.completions.create(
    model="gpt-4o-2024-05-13",
    messages=[{"role": "user", "content": "Return JSON: {city, population}"}]
)
print(resp.system_fingerprint)
print(resp.choices[0].message.content)

If you route through an OpenAI-compatible gateway such as n4n.ai, you can send a routing directive to lock the provider and model revision, bypassing automatic fallback while you diagnose. This isolates whether the default multi-provider route is masking a bad backend.

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "X-Route-Provider: openai" \
  -H "X-Route-Model: gpt-4o-2024-05-13" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Return JSON: {city, population}"}]}'

Compare the pinned call against the same call on your default route. If the pinned call is clean but the default route is broken, your fallback chain is hiding a degraded provider.

Verify success

You can reproduce the bad output on one path and get good output on the pinned path. That localizes the failure to prompt, model version, or provider.

Step 3: Check provider fallback and metering logs

Automatic fallback hides outages. If your gateway retries on 429 or 5xx, a silently degraded provider may be serving 80% of traffic while you watch the healthy one in manual tests. A provider can return 200 with truncated or low-quality text, and the fallback logic never triggers.

Pull per-token usage metering segmented by provider. A sudden drop in token volume from Provider A with a corresponding spike in Provider B indicates fallback fired.

{"provider":"openai","tokens":12000,"timestamp":"2024-06-01T10:00:00Z"}
{"provider":"anthropic","tokens":98000,"timestamp":"2024-06-01T10:00:00Z"}

If openai normally serves 50% and now serves 10%, investigate openai’s status independent of your code. Per-token metering is the only unbiased signal here; your own success metrics are polluted by the fallback.

Verify success

You know which provider absorbed the fallback and whether its quality is the cause. If both providers show equal degradation, the model version or your prompt is the culprit.

Step 4: Reproduce with a minimal controlled input

Strip your production prompt to the smallest case that still fails. This removes noise from retrieval augmentation, few-shot examples, and conversation history.

prompt = "Output a valid JSON object with keys: city, population. No prose."
bad = client.chat.completions.create(
    model="gpt-4o-2024-05-13",
    temperature=0,
    messages=[{"role": "user", "content": prompt}]
)
print(bad.choices[0].message.content)

Set temperature=0 to reduce sampling variance. If this minimal prompt fails, the model itself regressed. If it passes, bisect your prompt by adding components back one at a time: system message, then retrieved context, then few-shot examples.

Verify success

You have a 5-line reproduction that fails deterministically, or you proved the prompt edit from two days ago is the trigger. Either way, the search space is now small.

Step 5: Capture and diff raw responses

Log the full HTTP body, including usage and any provider-specific extensions. Diff the failing response against a known-good one from last week, stored in your eval archive.

import difflib
a = open("good.txt").read().splitlines()
b = open("bad.txt").read().splitlines()
print("\n".join(difflib.unified_diff(a, b, lineterm="")))

Look for truncated finish_reason, unexpected role changes, or dropped function_call arguments. These signal provider-side decoding changes. Pay attention to usage.completion_tokens: a sudden drop suggests the model is aborting early.

Verify success

You can point to the exact field that diverged. “The bad response has finish_reason: length and 50 fewer tokens” is a root cause; “it feels worse” is not.

Step 6: Apply a temporary mitigation

Do not rewrite your whole prompt under incident pressure. Freeze the model version, force a healthy provider, and enable provider cache-control hints to reduce variance from repeated calls. A cached response is at least consistent.

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "X-Cache-Control: max-age=3600" \
  -d '{"model":"gpt-4o-2024-05-13","messages":[{"role":"user","content":"Output JSON {city, population}"}]}'

If you must degrade gracefully, switch to a stricter parser or fall back to a rules engine for the affected intent. The goal is to stop the bleeding, not to ship a permanent fix during a war room.

Verify success

Contract failure rate drops below baseline within one deployment cycle. The pinned route serves production traffic until the postmortem completes.

Step 7: Write the postmortem and add a regression gate

A quality incident with no test is a recurring incident. Store the minimal reproduction from Step 4 as a unit test in your eval suite and run it on every model version bump.

def test_json_constraint():
    resp = client.chat.completions.create(
        model="gpt-4o-2024-05-13",
        temperature=0,
        messages=[{"role":"user","content":"Output JSON {city, population}"}]
    )
    content = resp.choices[0].message.content
    assert "city" in content and "population" in content

Schedule this against the live default route weekly. If a provider pushes a silent update, your CI catches it before production does. Include the system_fingerprint in the test output so you notice backend rotations.

Verify success

The test fails on the bad model version and passes on the pinned one; it runs in CI on a schedule and alerts on regression.

How to verify overall success

Your incident is resolved when three conditions hold: the contract failure rate returns to the pre-incident baseline for a full business day; the pinned model path and the default routed path produce equivalent quality on the golden set; and a postmortem doc lists the trigger, the layer (prompt/model/provider), and the test that now guards it.

Triaging degraded LLM response quality is iterative. The moment you stop measuring, the regression returns. Build the gauges before the next outage, not during it.