Building a content moderation test suite is the only way to trust a classifier in production. This tutorial walks through a pytest-based harness that measures accuracy against a labeled corpus, catches regressions when you swap models, and tunes score thresholds with real numbers.
Prerequisites
- Python 3.10 or newer
pytestandrequestsinstalled (pip install pytest requests)- An OpenAI-compatible moderation endpoint. OpenAI’s
/v1/moderationsworks; if you pointMODERATION_BASE_URLat n4n.ai’s OpenAI-compatible endpoint you also get automatic fallback when a provider is degraded. - A labeled dataset of texts and expected flag categories (we’ll bootstrap a small one)
Dataset design
A content moderation test suite lives or dies by the labels. Use JSONL with one text field and an expected list of category strings (empty list means clean).
{"text": "You are stupid and should leave", "expected": ["harassment"]}
{"text": "The meeting is at 3pm", "expected": []}
{"text": "I will kill you", "expected": ["violence"]}
{"text": "Buy cheap meds at spam.example.com", "expected": ["spam"]}Keep categories aligned with your endpoint’s taxonomy. OpenAI returns harassment, hate, self-harm, sexual, violence, spam (in some versions). Match exactly.
Client wrapper
Write a thin client that posts to /v1/moderations and returns flagged categories. Avoid the SDK; requests is clearer for testing.
# client.py
import os
import requests
BASE_URL = os.environ.get("MODERATION_BASE_URL", "https://api.openai.com/v1")
API_KEY = os.environ.get("MODERATION_API_KEY", "")
def moderate(text: str) -> list[str]:
resp = requests.post(
f"{BASE_URL}/moderations",
headers={"Authorization": f"Bearer {API_KEY}"},
json={"input": text, "model": "text-moderation-latest"},
timeout=10,
)
resp.raise_for_status()
result = resp.json()["results"][0]
return [cat for cat, flagged in result["categories"].items() if flagged]For score-based tuning, grab category_scores instead:
def moderate_scores(text: str) -> dict[str, float]:
resp = requests.post(
f"{BASE_URL}/moderations",
headers={"Authorization": f"Bearer {API_KEY}"},
json={"input": text, "model": "text-moderation-latest"},
timeout=10,
)
resp.raise_for_status()
return resp.json()["results"][0]["category_scores"]Parametric unit tests
pytest’s parametrize gives you one test per case. This isolates failures and shows exactly which string broke.
# test_moderation.py
import json
import pytest
from client import moderate
def _load_cases():
with open("dataset.jsonl") as f:
return [json.loads(line) for line in f if line.strip()]
CASES = _load_cases()
@pytest.mark.parametrize("case", CASES, ids=lambda c: c["text"][:25])
def test_case_flag_match(case):
predicted = set(moderate(case["text"]))
expected = set(case["expected"])
# strict equality for unit-level signal
assert predicted == expected, f"pred={predicted} exp={expected}"Run a first checkpoint:
pytest test_moderation.py -qExpected output (with our 4-line dataset, assuming the model flags correctly):
.... [100%]
4 passedIf a case fails, you see the exact text and mismatch.
Aggregate accuracy test
Unit tests are noisy. Add a suite-level test that computes precision/recall across all categories.
ALL_CATS = ["harassment", "hate", "self-harm", "sexual", "violence", "spam"]
def test_accuracy_thresholds():
tp = fp = fn = 0
for case in CASES:
pred = set(moderate(case["text"]))
exp = set(case["expected"])
for cat in ALL_CATS:
if cat in exp and cat in pred: tp += 1
elif cat not in exp and cat in pred: fp += 1
elif cat in exp and cat not in pred: fn += 1
precision = tp / (tp + fp) if tp + fp else 1.0
recall = tp / (tp + fn) if tp + fn else 1.0
assert precision >= 0.85, f"Precision {precision:.2f} < 0.85"
assert recall >= 0.80, f"Recall {recall:.2f} < 0.80"This is the core of a content moderation test suite: it fails the build if the classifier drifts below agreed guards.
Threshold tuning with scores
Default category booleans use a fixed cutoff (usually 0.5). You can do better by sweeping moderate_scores and picking a threshold per category that maximizes F1 on your dataset.
def tune_thresholds(cases, cats, grid=(0.3, 0.4, 0.5, 0.6, 0.7)):
best = {}
for cat in cats:
best_f1 = -1
for thr in grid:
tp=fp=fn=0
for c in cases:
score = moderate_scores(c["text"]).get(cat, 0.0)
pred = score >= thr
exp = cat in c["expected"]
if exp and pred: tp+=1
elif not exp and pred: fp+=1
elif exp and not pred: fn+=1
prec = tp/(tp+fp) if tp+fp else 1
rec = tp/(tp+fn) if tp+fn else 1
f1 = 2*prec*rec/(prec+rec) if prec+rec else 0
if f1 > best_f1:
best_f1, best[cat] = f1, thr
return bestRun it in a script, not a test, to print recommendations:
python -c "from test_moderation import CASES, tune_thresholds, ALL_CATS; print(tune_thresholds(CASES, ALL_CATS))"Output might be {'harassment': 0.4, 'violence': 0.5, ...}. Feed those back into client.py as a threshold map.
Regression baseline
A content moderation test suite must detect performance drops after model upgrades. Store last run’s metrics in baseline.json and compare.
import json
def test_no_regression():
# assume metrics computed as above
current = {"precision": precision, "recall": recall}
try:
with open("baseline.json") as f:
base = json.load(f)
except FileNotFoundError:
with open("baseline.json", "w") as f:
json.dump(current, f)
return
drop = base["recall"] - current["recall"]
assert drop <= 0.02, f"Recall dropped {drop:.2f} > 0.02"Commit baseline.json only after a human reviews a good run.
Running the full suite
export MODERATION_API_KEY=sk-yourkey
export MODERATION_BASE_URL=https://api.openai.com/v1
pytest -qExpected final output:
....... [100%]
7 passedIf you later swap MODERATION_BASE_URL to a gateway, the same suite validates the new backend without code changes.
What to add next
- Expand dataset with edge cases: code-switching, obfuscation (
k1ll), benign medical text. - Add latency assertions (
assert resp.elapsed < 300ms). - Wire the suite into CI; block merges on recall regression.
- For per-token cost tracking, n4n.ai meters usage and forwards cache-control hints, so you can bill test runs accurately.
That’s a complete, runnable content moderation test suite. Build the dataset first, then trust the numbers.