How content moderation models classify harmful text

Content moderation models are classifiers that map text (and increasingly multimodal inputs) to a taxonomy of harm categories — hate speech, violence, sexual content, self-harm, PII, and policy-specific violations — producing structured labels and confidence scores that downstream systems use to block, flag, route, or log. Understanding how content moderation models work means understanding the interplay between taxonomy design, model architecture, calibration, and the operational thresholds that turn probabilities into enforcement decisions. This post walks through each layer with the specificity an engineer needs to integrate, evaluate, or build these systems.

Taxonomy design drives everything

Before a single weight is trained, the taxonomy defines what the model must distinguish. A flat label set (safe / unsafe) is useless for production; you need hierarchical, mutually exclusive, and collectively exhaustive categories that map to policy actions.

A typical hierarchy looks like:

harmful_content/
├── hate_speech/
│   ├── race_ethnicity
│   ├── religion
│   ├── sexual_orientation
│   ├── gender_identity
│   └── disability
├── harassment/
│   ├── targeted_insult
│   ├── threat
│   └── doxxing
├── violence_physical/
│   ├── graphic_violence
│   ├── incitement
│   └── weapons_instructions
├── sexual_content/
│   ├── explicit_sexual
│   ├── minors_sexualized
│   └── non_consensual_intimate
├── self_harm/
│   ├── suicide
│   ├── eating_disorder
│   └── self_injury
├── illegal_acts/
│   ├── drug_manufacturing
│   ├── fraud_scam
│   └── cybercrime_instructions
└── pii/
    ├── ssn
    ├── credit_card
    ├── api_key
    └── address_phone

Each leaf node should correspond to a distinct enforcement action: block, shadow-ban, human review, log-only, or rewrite. If two categories always trigger the same action, merge them. If a single category needs different actions by context (e.g., medical vs. erotic sexual content), split it.

Taxonomy versioning is non-negotiable. Label your data with taxonomy_version: "2024-03-15" and store the mapping in your model registry. When policy changes, you retrain or fine-tune against the new version — never silently relabel in place.

Model architectures in production

Three architectural patterns dominate. Most production systems ensemble them.

1. Encoder-only transformers (BERT-family)

Fine-tuned roberta-large, deberta-v3-large, or domain-specific variants (e.g., hatebert, toxic-bert) remain the workhorse for single-label and multi-label classification. They’re fast, well-understood, and easy to calibrate.

# Minimal inference wrapper
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

class ModerationClassifier:
    def __init__(self, model_id: str, taxonomy: list[str], threshold: float = 0.5):
        self.tokenizer = AutoTokenizer.from_pretrained(model_id)
        self.model = AutoModelForSequenceClassification.from_pretrained(model_id)
        self.taxonomy = taxonomy
        self.threshold = threshold
        self.model.eval()

    @torch.inference_mode()
    def predict(self, texts: list[str]) -> list[dict]:
        inputs = self.tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
        logits = self.model(**inputs).logits
        probs = torch.sigmoid(logits).cpu().numpy()
        results = []
        for prob in probs:
            labels = [self.taxonomy[i] for i, p in enumerate(prob) if p >= self.threshold]
            scores = {self.taxonomy[i]: float(p) for i, p in enumerate(prob)}
            results.append({"labels": labels, "scores": scores})
        return results

Key operational details:

Max length 512 truncates long context. For documents, chunk with overlap and aggregate (max-pool or mean-pool per label).
Batch inference is mandatory. Pad to the batch’s max length, not 512, to save compute.
ONNX / TensorRT / OpenVINO export cuts latency 3-5x. Quantize to INT8 after verifying calibration drift < 0.5% F1.

2. Decoder-only LLMs as classifiers

Prompting a 7B-70B model (Llama-3, Mistral, Qwen) with a structured taxonomy works surprisingly well for nuanced categories — especially “context-dependent” ones like harassment vs. quoting harassment, or medical vs. explicit sexual content.

{
  "system": "You are a content moderation classifier. Output ONLY valid JSON matching the schema.",
  "user": "Classify the following text into zero or more categories from this taxonomy:\n- hate_speech.race_ethnicity\n- harassment.targeted_insult\n- violence_physical.graphic_violence\n- sexual_content.explicit_sexual\n- self_harm.suicide\n- pii.ssn\n\nText: \"{input_text}\"\n\nReturn: {\"labels\": [\"category.path\"], \"reasoning\": \"brief justification\"}"
}

Trade-offs:

Latency: 100-500ms vs. 5-20ms for encoder models. Use for ambiguous cases only (cascade).
Cost: API or self-hosted GPU. Cache aggressively — identical inputs should hit a lookup table.
Calibration: LLMs are poorly calibrated out of the box. You need a temperature-scaling head or a small calibration set (500-1k examples) to map logits to reliable probabilities.

3. Lightweight heads on frozen embeddings

For high-throughput, low-latency paths (e.g., every chat message at 10k QPS), freeze a small encoder (DistilBERT, MiniLM-L6, or a custom 4-layer transformer) and train only a multi-label classification head. Latency drops to 1-3ms on CPU.

# DistilBERT + linear head, ~1.5ms CPU inference
from sentence_transformers import SentenceTransformer
import torch.nn as nn

class TinyModerationHead(nn.Module):
    def __init__(self, embed_dim: int, num_labels: int):
        super().__init__()
        self.encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
        for p in self.encoder.parameters():
            p.requires_grad = False
        self.head = nn.Linear(embed_dim, num_labels)

    def forward(self, texts: list[str]) -> torch.Tensor:
        emb = self.encoder.encode(texts, convert_to_tensor=True, normalize_embeddings=True)
        return torch.sigmoid(self.head(emb))

This pattern shines when you need per-token or span-level labels (e.g., highlight the exact PII span). Add a token-classification head on top of the same frozen encoder.

Calibration and threshold selection

Raw model outputs are not probabilities. A 0.9 score on “hate_speech.race_ethnicity” does not mean 90% of such predictions are correct. You must calibrate.

Temperature scaling (post-hoc)

from sklearn.isotonic import IsotonicRegression
import numpy as np

def calibrate_per_label(logits: np.ndarray, labels: np.ndarray) -> list[IsotonicRegression]:
    """Fit isotonic regression per label on a held-out calibration set."""
    calibrators = []
    probs = 1 / (1 + np.exp(-logits))  # sigmoid
    for i in range(logits.shape[1]):
        ir = IsotonicRegression(out_of_bounds="clip")
        ir.fit(probs[:, i], labels[:, i])
        calibrators.append(ir)
    return calibrators

def apply_calibration(probs: np.ndarray, calibrators: list) -> np.ndarray:
    return np.column_stack([calibrators[i].transform(probs[:, i]) for i in range(probs.shape[1])])

Use isotonic regression (non-parametric) over Platt scaling (sigmoid) — it handles the long-tail miscalibration common in moderation data. Reserve 5-10% of your labeled data only for calibration; never touch it during training or threshold tuning.

Thresholds per label, not global

A single global threshold (e.g., 0.5) is a bug. Each category has different prevalence, cost of false positives, and cost of false negatives. Set thresholds by optimizing a utility function on a validation set:

def find_optimal_thresholds(
    probs: np.ndarray,
    labels: np.ndarray,
    fp_cost: np.ndarray,   # shape (num_labels,)
    fn_cost: np.ndarray,
) -> np.ndarray:
    thresholds = np.zeros(probs.shape[1])
    for i in range(probs.shape[1]):
        # Sweep thresholds, compute expected cost
        best_thresh, best_cost = 0.5, float("inf")
        for t in np.linspace(0.01, 0.99, 99):
            preds = (probs[:, i] >= t).astype(int)
            fp = np.sum((preds == 1) & (labels[:, i] == 0))
            fn = np.sum((preds == 0) & (labels[:, i] == 1))
            cost = fp * fp_cost[i] + fn * fn_cost[i]
            if cost < best_cost:
                best_cost, best_thresh = cost, t
        thresholds[i] = best_thresh
    return thresholds

Typical cost asymmetries:

CSAM / minors_sexualized: FN cost >> FP cost (threshold ~0.15)
PII: FP cost high (user friction), FN cost regulatory (threshold ~0.7)
Hate speech: Balanced but context-dependent (threshold ~0.4-0.6)
Spam / promo: FP cost high (threshold ~0.8)

Store thresholds in config, versioned with the model. thresholds_v3.yaml deploys with model_v3.onnx.

Cascading architecture for production

No single model satisfies latency, cost, and quality simultaneously. Cascade:

Request
  │
  ├─► Stage 1: Tiny head (CPU, 2ms) ──► "clearly safe" (max_prob < 0.15) ──► ALLOW
  │
  ├─► Stage 2: Encoder (GPU/ONNX, 15ms) ──► confident predictions (max_prob > 0.85 or < 0.25) ──► DECIDE
  │
  └─► Stage 3: LLM judge (GPU, 200ms) ──► ambiguous zone ──► FINAL DECISION + reasoning

The cascade reduces average latency by 10-20x while keeping LLM calls under 5% of traffic. Implement as a single async pipeline with a shared request context:

async def moderate(text: str, context: RequestContext) -> ModerationResult:
    # Stage 1
    tiny_probs = await tiny_model.predict([text])
    if tiny_probs.max() < 0.15:
        return ModerationResult(action="allow", stage="tiny", scores=tiny_probs)

    # Stage 2
    encoder_probs = await encoder_model.predict([text])
    max_p = encoder_probs.max()
    if max_p > 0.85 or max_p < 0.25:
        return ModerationResult(action=decide(encoder_probs), stage="encoder", scores=encoder_probs)

    # Stage 3
    llm_result = await llm_judge.classify(text, taxonomy)
    return ModerationResult(action=decide(llm_result), stage="llm", scores=llm_result.probs, reasoning=llm_result.reasoning)

Log every stage’s output. You need this for:

Drift detection: Compare Stage 1 vs Stage 2 label agreement over time.
Cost accounting: Track LLM token spend per 1k requests.
Human review sampling: Prioritize cases where stages disagree.

Multimodal and multilingual considerations

Text-only moderation misses 30-50% of policy violations on platforms with images, video, or audio. Two practical patterns:

Early fusion (single model)

Train a vision-language model (CLIP, SigLIP, Llava, Qwen-VL) with a classification head on concatenated [image_embedding; text_embedding]. Simpler deployment, but requires massive labeled multimodal data.

Late fusion (ensemble)

Run independent image and text classifiers, then combine:

def late_fusion(image_probs: np.ndarray, text_probs: np.ndarray, weights: dict) -> np.ndarray:
    # weights: {"image": 0.6, "text": 0.4} per label, learned on validation
    combined = np.zeros_like(image_probs)
    for i, label in enumerate(taxonomy):
        w_img = weights[label].get("image", 0.5)
        w_txt = weights[label].get("text", 0.5)
        combined[:, i] = w_img * image_probs[:, i] + w_txt * text_probs[:, i]
    return combined

Late fusion lets you reuse existing best-in-class unimodal models and update them independently. For multilingual text, use a single multilingual encoder (XLM-RoBERTa, mDeBERTa, or a multilingual E5/MiniLM) rather than per-language models — it handles code-switching and low-resource languages without a language-ID router.

Concrete example: PII detection with span extraction

PII is a category where classification alone fails — you need to locate the entity to redact or block. Token classification (NER-style) on a frozen encoder:

# Training: token-level labels (B-IO scheme)
# "My SSN is 123-45-6789" → [O, O, O, B-SSN, I-SSN, I-SSN]

from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

model = AutoModelForTokenClassification.from_pretrained(
    "microsoft/mdeberta-v3-base",
    num_labels=len(label2id),
    id2label=id2label,
    label2id=label2id,
)

training_args = TrainingArguments(
    output_dir="pii-span-model",
    per_device_train_batch_size=32,
    learning_rate=2e-5,
    num_train_epochs=3,
    weight_decay=0.01,
    evaluation_strategy="steps",
    eval_steps=500,
    save_strategy="steps",
    load_best_model_at_end=True,
    metric_for_best_model="f1",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    tokenizer=tokenizer,
    data_collator=DataCollatorForTokenClassification(tokenizer),
    compute_metrics=compute_metrics,  # seqeval F1 per entity type
)
trainer.train()

Inference returns spans: {"entity": "SSN", "start": 11, "end": 22, "score": 0.98, "text": "123-45-6789"}. Downstream, you can:

Redact: Replace with [SSN_REDACTED]
Block: Reject the message
Route: Send to compliance queue
Hash: Store salted hash for repeat-detection without storing raw PII

This pattern generalizes to secrets detection (API keys, JWTs, connection strings) — just expand the label set and add regex pre-filters for high-recall candidates.

Common misconceptions

“One model to rule them all”

A single multi-label classifier cannot simultaneously optimize for CSAM (extreme recall), PII (high precision + span extraction), and nuanced harassment (contextual reasoning). The cascade exists because the operational requirements differ per category. Build specialist heads, route intelligently.

“High AUC means production ready”

AUC measures ranking quality across all thresholds. Production runs at one threshold per label. A model with AUC 0.98 can have 40% precision at your operating threshold if the positive class is 0.1% prevalence. Always evaluate at your deployed thresholds on a current holdout set.

“Human labels are ground truth”

Moderation labels have inter-annotator agreement (IAA) of 0.7-0.85 Cohen’s kappa on complex categories. Your model cannot exceed the noise ceiling of your labels. Invest in:

Adjudication workflows: Senior annotators resolve disagreements
Policy playbooks: Decision trees with examples, not vague guidelines
Label quality metrics: Track IAA per category per week; investigate drops

“We’ll just use GPT-4o / Claude as the moderator”

LLM-as-judge works for evaluation and ambiguous cases. It fails as a primary filter because:

Latency variance: 200ms → 5s under load
Non-determinism: Same input, different labels across calls
No calibration: You cannot set a threshold on “high/medium/low”
Cost: $2-15/M tokens vs. $0.0001/M for a distilled encoder

Use LLMs to generate training data for your specialist models (few-shot prompting with verified labels), not as the serving path.

“False positives are just a UX problem”

False positives on hate speech or harassment silence marginalized users disproportionately — the very groups the policies aim to protect. Measure disparate impact by demographic inference (language, dialect, name embeddings) on your false positive set. If FPR for AAVE dialect is 3x standard English, your model is broken, not your threshold.

Evaluation that matters

Don’t report accuracy. Report per-label at operating threshold:

Metric	Why
Precision @ threshold	User-facing: how often is a block correct?
Recall @ threshold	Safety: how much harm slips through?
FPR @ threshold	Volume: how many safe items get human review?
Latency p50/p99	SLA compliance
Calibration error (ECE)	Trustworthy probabilities for cascading
Disparate FPR by dialect/group	Fairness / legal risk

Track these on a rolling 7-day evaluation set sampled from live traffic (with PII stripped), not a static test set. Data drift in moderation is fast — new slang, new attack vectors, new platform features. If your eval set is 3 months old, your metrics are fiction.

Operational checklist for a new category

Define the leaf node in taxonomy with enforcement action
Collect 2k-5k labeled examples (active learning: model uncertainty + human review)
Measure IAA; if < 0.75, refine playbook and relabel
Train specialist head on frozen encoder; calibrate on held-out 1k
Set threshold via cost matrix; document FP/FN cost assumptions
Add to cascade with routing rules (which stage, fallback logic)
Deploy shadow mode (log only, no enforcement) for 2 weeks
Compare shadow vs. production on disparate impact, drift, cost
Enable enforcement with gradual rollout (1% → 10% → 100%)
Schedule retraining monthly or when drift alert fires

Closing thought

Content moderation is not a model problem — it’s a system problem. The model is the easiest component to swap. The hard parts are taxonomy governance, label quality, calibration discipline, cascade orchestration, and the feedback loops that keep the system honest as language and abuse evolve. Build the infrastructure around the model first; the model itself is interchangeable.