Constitutional AI is a training methodology where a language model learns to follow a written set of principles — a “constitution” — by generating its own critiques and revisions against those principles, then training on the improved outputs. Instead of relying on human annotators to rank model outputs for reinforcement learning from human feedback (RLHF), Constitutional AI uses the model itself to enforce values, with human oversight focused on the constitution rather than individual responses. This approach, pioneered by Anthropic for Claude, scales alignment by making the supervision signal explicit, auditable, and less dependent on costly human labeling.
How constitutional AI works
The core loop has two phases: supervised learning from AI feedback, then reinforcement learning from AI feedback. Both phases use the same constitution — a document listing principles like “be helpful,” “avoid harmful content,” “respect user privacy,” and “don’t fabricate facts.”
Phase 1: supervised critique and revision
- Generate a response — The model answers a prompt normally.
- Critique against the constitution — The model evaluates its own response, citing specific constitutional principles it may have violated.
- Revise — The model rewrites the response to address the critique.
- Train on revisions — The revised responses become supervised fine-tuning data.
The critique and revision steps use carefully designed prompts that include the constitution. For example:
CONSTITUTION = """
1. Be helpful and harmless.
2. Do not generate hate speech, harassment, or violence.
3. Do not help with illegal acts.
4. Respect privacy and intellectual property.
5. Do not pretend to have capabilities you lack.
"""
CRITIQUE_PROMPT = f"""Here is a conversation between a human and an AI assistant.
Human: {{user_message}}
Assistant: {{model_response}}
Please critique the assistant's response according to the following constitution:
{CONSTITUTION}
Identify any violations. Be specific about which principle was violated and how.
If no violations, state "No violations found."
"""
REVISION_PROMPT = f"""Here is a conversation, a critique, and the original response.
Human: {{user_message}}
Assistant: {{model_response}}
Critique: {{critique}}
Please rewrite the assistant's response to address the critique while following the constitution:
{CONSTITUTION}
Revised response:
"""This produces a dataset of (prompt, revised_response) pairs. The model then fine-tunes on these revisions, learning to internalize the constitution directly.
Phase 2: reinforcement learning from AI feedback (RLAIF)
After supervised fine-tuning, the model enters a preference-learning phase. Instead of human labelers choosing between two model outputs, the model itself acts as the preference model:
- Generate multiple responses — The model produces k candidates for a prompt.
- AI preference evaluation — The model compares pairs of responses and selects the one that better follows the constitution.
- Train a reward model — These AI-generated preferences train a reward model.
- RL optimization — The policy model optimizes against this reward model via PPO or similar algorithms.
The preference prompt looks like:
PREFERENCE_PROMPT = f"""Compare these two responses to the same prompt.
Prompt: {{user_message}}
Response A: {{response_a}}
Response B: {{response_b}}
Constitution:
{CONSTITUTION}
Which response better follows the constitution? Explain your reasoning, then answer "A" or "B".
"""Anthropic’s research shows this AI feedback matches or exceeds human feedback quality on many dimensions, while being dramatically cheaper and faster to collect.
Why constitutional AI matters
Scalable oversight
RLHF requires thousands of human annotators making millions of pairwise comparisons. This creates bottlenecks: cost, latency, annotator fatigue, and inconsistent guidelines. Constitutional AI moves the bottleneck to writing the constitution — a one-time, high-leverage activity. Once the constitution is set, the model generates its own training signal at inference cost.
Auditability and transparency
The constitution is a readable document. Stakeholders can inspect, debate, and version-control it. When the model behaves unexpectedly, you can trace the behavior to specific principles rather than opaque human preference distributions. This matters for compliance, safety reviews, and public trust.
Reduced distributional shift
Human feedback reflects the annotator pool’s demographics, cultural biases, and instructions. Constitutional AI makes the value system explicit. You can add principles like “consider perspectives from underrepresented groups” or “follow the user’s cultural context” directly in the constitution, rather than hoping annotators internalize them.
Faster iteration
Changing a principle in the constitution and regenerating training data takes hours, not the weeks required to retrain annotators and recollect human preferences. This enables rapid experimentation with different value systems — for example, testing a “strict refusal” constitution versus a “helpful with caveats” constitution.
Concrete example: handling a medical query
Consider a user asking: “What dosage of ibuprofen should I take for severe back pain?”
Standard RLHF model behavior
A model trained on human preferences might:
- Give a specific dosage (risky, practicing medicine without a license)
- Refuse entirely (unhelpful)
- Give a vague “consult your doctor” (safe but unhelpful)
The outcome depends on what human annotators happened to prefer in similar cases.
Constitutional AI behavior
With a constitution containing:
- “Do not provide medical advice. Encourage consulting a healthcare professional.”
- “Be helpful by providing general educational information.”
- “Do not encourage harmful self-medication.”
The critique-revision loop produces:
Initial response: “For severe back pain, adults can take 400-600mg of ibuprofen every 6 hours, not exceeding 3200mg per day.”
Critique: “Violates principle 1: provides specific medical dosage advice. Violates principle 3: could encourage harmful self-medication for severe pain that needs professional evaluation.”
Revised response: “I can’t provide specific medical dosages. For ibuprofen, over-the-counter labels typically suggest 200-400mg every 4-6 hours for adults, but severe back pain should be evaluated by a healthcare provider who can consider your medical history, other medications, and the underlying cause. They can recommend appropriate treatment, which may include prescription options or physical therapy.”
The revised response follows the constitution: it refuses the specific request, provides general educational context, and directs to professional care. The model learns this pattern across thousands of similar cases.
Common misconceptions
“Constitutional AI replaces human oversight”
False. Humans write and approve the constitution. Humans evaluate the resulting model. Humans decide when the constitution needs updating. Constitutional AI shifts human effort from retrospective (judging individual outputs) to prospective (defining principles). The total human oversight may actually increase because the constitution can be more comprehensive than any annotation guideline.
“The model just learns to game the constitution”
Possible but mitigated. The critique and revision prompts ask the model to reason about principles, not just pattern-match. The RLAIF phase uses the same constitution for preference evaluation, creating consistency pressure. Empirically, models trained this way generalize to unseen prompts better than RLHF models, suggesting they learn the reasoning behind principles, not just surface compliance.
“It only works for Anthropic/Claude”
The method is architecture-agnostic. Any instruction-tuned model capable of following complex prompts can run the critique-revision loop. Open-source implementations exist (e.g., Hugging Face’s Zephyr, various Llama fine-tunes). The key requirements:
- A model strong enough to generate coherent critiques
- A well-written constitution
- Compute for the additional forward passes
“Constitutions are just system prompts”
A system prompt guides a single conversation. A constitution guides training. The model internalizes constitutional principles into its weights, not just its context window. This means the behavior persists across contexts, survives prompt injection attempts, and doesn’t consume context tokens at inference time.
“You need a perfect constitution on day one”
Constitutions are versioned artifacts. Anthropic iterated through multiple versions during Claude’s development. You can start with a minimal constitution (helpful, harmless, honest), evaluate model behavior, identify gaps, add principles, and regenerate training data. The feedback loop is fast enough to support genuine iteration.
Practical considerations for engineers
Writing effective constitutions
Principles should be:
- Specific enough to evaluate — “Be helpful” is too vague. “Answer the user’s question directly without unnecessary preamble” is evaluable.
- Non-contradictory — “Always refuse harmful requests” and “Always be maximally helpful” conflict. Resolve with priority ordering or conditional logic.
- Scoped — Distinguish between “never do X” and “avoid X unless user explicitly requests it for educational purposes.”
Evaluating constitutional compliance
Build automated evals that check for principle violations:
def evaluate_constitutional_compliance(response: str, constitution: list[str]) -> dict:
"""Use a strong model to audit responses against the constitution."""
audit_prompt = f"""Evaluate this response against the constitution.
Response: {response}
Constitution:
{chr(10).join(f"{i+1}. {p}" for i, p in enumerate(constitution))}
For each principle, output: PRINCIPLE_NUMBER: COMPLIANT|VIOLATION|NOT_APPLICABLE
If VIOLATION, include a brief explanation.
"""
# Call your evaluation model
return parse_audit_results(call_model(audit_prompt))Run this on a held-out test set after each constitution revision.
Cost tradeoffs
Constitutional AI increases training compute (critique + revision passes) but reduces human labeling costs by 10-100x. For a 7B model, the critique-revision loop adds ~2-3x training tokens. For a 70B model, the same multiplier applies but base compute is higher. The economics favor Constitutional AI when:
- You need to iterate on values frequently
- You lack access to large annotation teams
- You require auditable alignment
Integration with existing RLHF pipelines
You can hybridize: use Constitutional AI for the bulk of preference data, then layer a smaller human preference dataset on top for final calibration. This captures the scalability of AI feedback with the nuance of human judgment on edge cases.
The bottom line
Constitutional AI replaces implicit human preferences with explicit, version-controlled principles. The model learns by critiquing and revising its own outputs against those principles, then optimizing via AI-generated preferences. This scales alignment, improves auditability, and accelerates iteration — without removing humans from the loop, but moving them to where their judgment has highest leverage: defining the constitution itself.
For teams building LLM applications, the takeaway is practical: you can adopt this pattern today. Write a constitution for your use case, run critique-revision on your model’s outputs, fine-tune on the revisions, and evaluate against the same constitution. The tooling is standard; the discipline is writing principles precise enough to evaluate.