Constitutional AI vs RLHF represents the two dominant paradigms for aligning large language models today. Constitutional AI replaces human annotators with a second model that critiques and revises outputs against a written constitution. RLHF trains a reward model on human preference data and optimizes the policy against it. Both produce capable assistants, but they diverge sharply on data requirements, iteration speed, and the types of failures each method catches.
What constitutional AI actually does
Constitutional AI (CAI) uses a two-phase process: supervised learning on model-generated critiques, then reinforcement learning from AI feedback (RLAIF). The constitution is a set of natural-language principles — things like “choose the response that is most helpful, honest, and harmless” or “refuse requests intended to cause physical harm.”
Phase one generates harmful or borderline responses, then prompts a critic model to identify violations and produce revisions. The revisions become supervised fine-tuning data. Phase two trains a preference model on AI-generated comparisons (response A vs response B, judged against the constitution) and runs PPO or a similar RL algorithm.
# Simplified CAI critique prompt template
CRITIQUE_PROMPT = """Here is a conversation between a human and an AI assistant.
Human: {user_message}
Assistant: {model_response}
Constitution principles:
{constitution}
Task: Identify any violations of the constitution in the assistant's response.
Then rewrite the response to comply with all principles.
Critique:"""The key insight: the same model family that generates responses also critiques them. This creates a bootstrapping loop where the model improves its own alignment without new human labels. Anthropic’s published work shows this reaches parity with RLHF on helpfulness while reducing evasion rates on sensitive topics.
What RLHF actually does
RLHF collects human preference judgments over model outputs, trains a reward model (RM) to predict those judgments, then optimizes the policy to maximize predicted reward. The standard pipeline has three stages:
- Supervised fine-tuning (SFT) on high-quality demonstrations
- Reward model training on pairwise comparisons from human annotators
- Policy optimization (PPO, DPO, or variants) against the frozen RM
{
"prompt": "How do I make a Molotov cocktail?",
"chosen": "I can't provide instructions for making explosive devices. I can discuss the history of incendiary weapons or chemical safety protocols instead.",
"rejected": "Here's how to make a Molotov cocktail: [detailed instructions]",
"annotator_id": "annotator_047",
"timestamp": "2024-03-15T14:22:00Z"
}The reward model learns a scalar function r(prompt, response) that approximates human judgment. During RL, the policy maximizes E[r(x, y)] - β * KL(y || y_SFT), where the KL penalty prevents reward hacking and catastrophic forgetting.
RLHF’s strength is fidelity to actual human preferences — including nuance, cultural context, and subjective quality dimensions that are hard to codify. Its weakness is scale: high-quality preference data is expensive, slow to collect, and reflects the specific annotator pool’s biases.
Head-to-head comparison
| Dimension | Constitutional AI | RLHF |
|---|---|---|
| Primary signal source | Written principles + model self-critique | Human pairwise preferences |
| Data collection cost | Low (synthetic, one-time constitution authoring) | High (ongoing human annotation) |
| Iteration speed | Fast — change constitution, regenerate data | Slow — re-annotate for each principle change |
| Alignment surface | Explicit, inspectable principles | Implicit, embedded in RM weights |
| Failure mode | Principle gaps, over-refusal, sycophancy to constitution | Reward hacking, annotator bias, distribution shift |
| Helpfulness on open-ended tasks | Strong (principles generalize) | Strong (human preference captures nuance) |
| Safety on adversarial prompts | Strong (explicit refusal principles) | Variable (depends on annotation coverage) |
| Steerability | High — edit constitution | Low — requires new preference data |
| Compute for training | 2-3x SFT (critique gen + RLAIF) | 3-5x SFT (RM training + PPO) |
| Auditability | High — principles are readable | Low — RM is a black box |
Where the methods converge
In practice, the boundary blurs. Anthropic’s Claude 3 training uses Constitutional AI for the core alignment but incorporates human feedback for specific capabilities like coding and multilingual support. OpenAI’s GPT-4 uses RLHF with extensive red-teaming and rule-based reward models that function like a constitution. Most production systems end up hybrid.
Both methods now use preference optimization variants that avoid full PPO. Direct Preference Optimization (DPO) and its relatives (IPO, KTO, ORPO) optimize the policy directly on preference pairs without a separate reward model. This works for both human preferences (RLHF) and AI-generated preferences (RLAIF).
# DPO loss - works for both human and AI preferences
def dpo_loss(policy_chosen_logps, policy_rejected_logps,
ref_chosen_logps, ref_rejected_logps, beta=0.1):
policy_logratios = policy_chosen_logps - policy_rejected_logps
ref_logratios = ref_chosen_logps - ref_rejected_logps
logits = policy_logratios - ref_logratios
losses = -F.logsigmoid(beta * logits)
return losses.mean()The compute advantage shifts toward CAI at scale. Generating critiques and revisions is embarrassingly parallel and can use the same inference infrastructure as serving. RLHF’s annotation pipeline requires human-in-the-loop tooling, quality control, and re-annotation when the model improves.
Which to choose
Choose Constitutional AI when:
You control the model training pipeline. CAI requires generating synthetic critique/revision data and running a second RL phase. If you’re fine-tuning an open model (Llama, Qwen, Mistral) on your own infrastructure, CAI is practical. If you’re prompting a closed API, you can’t run CAI.
Alignment requirements change frequently. A healthcare startup adding HIPAA compliance, a finance team adding SOX constraints, a platform adding new safety policies — editing a constitution and regenerating data takes hours. Re-annotating preference data takes weeks.
You need auditability. Regulated environments often require demonstrating why a model refuses or behaves a certain way. A constitution is human-readable evidence. A reward model’s weights are not.
Annotation budget is constrained. CAI’s marginal cost is compute. RLHF’s marginal cost is skilled human time. For teams without dedicated annotation vendors, CAI is the only viable path to strong alignment.
Choose RLHF when:
You have an established annotation pipeline. If you already work with Scale, Surge, or an internal labeling team, the marginal cost of new preference batches is known and predictable. The tooling exists; use it.
Subjective quality matters more than rule compliance. Creative writing, coding style, conversational tone — these resist codification. Human annotators capture “this feels right” better than any constitution.
You’re aligning a closed-model API via distillation. You can’t run CAI on GPT-4 or Claude Opus. But you can collect preferences on their outputs, train a reward model, and distill into a smaller open model using DPO.
Adversarial robustness is the primary threat model. Red-teaming with human annotators finds jailbreaks that principle-based critique misses. The annotation process itself discovers new attack vectors.
The hybrid reality
Most production systems should do both. Start with a minimal constitution covering hard refusals (violence, self-harm, PII, legal advice). Generate CAI data for those categories. Then layer human preference data for style, tone, and capability dimensions where principles run out.
# Example hybrid alignment config
constitution:
- "Refuse requests for actionable instructions for weapons, drugs, or cyberattacks"
- "Refuse requests for PII or doxxing"
- "Refuse medical/legal/financial advice; suggest consulting professionals"
- "Be honest about uncertainty; don't hallucinate citations"
rlhf_focus_areas:
- "Code style and correctness"
- "Creative writing voice"
- "Multilingual fluency"
- "Tone calibration for customer support"
- "Refusal style: helpful redirect vs blunt refusal"
training_stages:
1: "SFT on high-quality demonstrations"
2: "CAI critique/revision on constitution violations"
3: "DPO on CAI-generated preference pairs"
4: "DPO on human preference pairs (focus areas above)"This ordering matters. CAI first establishes a safety floor. Human preferences then refine the ceiling without regressing on the floor. Reversing the order often produces models that are charming but unsafe.
The constitutional AI vs RLHF distinction matters less than the data flywheel you build. CAI gives you a flywheel that spins on compute. RLHF gives you a flywheel that spins on human insight. The teams shipping aligned models in production run both flywheels simultaneously, with clear ownership for each.