Constitutional AI replaces most human preference labels with model self-critique against a written principles list. The policy becomes editable text instead of implicit labelling.
Imagine teaching someone manners. RLHF is like having a tutor sit with the student and grade every conversation: harsh, slow, and the tutor's taste leaks into every grade. Constitutional AI is like giving the student a small written rulebook and asking them to grade their own conversations against it, then revise. A judge model then compares pairs of answers using the same rulebook. The rulebook is short and human-readable, so when the rules change, you do not need to retrain the tutor. You edit a page. The student still practices and learns the same skill of being helpful and harmless; the change is in who holds the rubric and how visible it is.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Constitutional AI is a well-known example of what changes when you replace human preference labels with model-generated ones. The popular framing ('CAI is RLHF with AI labels') misses the more interesting structural difference. The shift is not just who labels the preferences. It is also where the safety policy lives, how it is reviewed, and how quickly it can change.
RLHF stores its safety policy implicitly in a reward model's weights, learned from many human preference labels. Constitutional AI stores it as a few pages of text plus the critic model that knows how to apply them. One you audit by inspecting outputs; the other you audit by reading the document.
The two-stage training loop in detail
Stage one: SL-CAI (supervised learning, constitutional)
The model is prompted with a potentially harmful or ambiguous question. It generates an initial response. Then a second prompt feeds the model its own response along with a randomly sampled principle from the constitution, and asks the model to critique whether the response violates that principle. A third prompt asks the model to rewrite the response addressing the critique.
The (original prompt, final rewritten response) pair becomes a supervised fine-tuning example. After thousands of such pairs across the prompt distribution, the base model is fine-tuned on this dataset. The model now has internalised the revision behaviour: when it sees a similar prompt, it tends to produce a revised-style answer directly.
Stage two: RL-CAI / RLAIF (reinforcement learning from AI feedback)
The SL-CAI model generates two candidate responses to the same prompt. A feedback model (in the paper, a pretrained language model) is asked to pick the one that better satisfies the constitution. This is run at scale: hundreds of thousands of prompts, many candidate pairs each.
The (prompt, preferred response, dispreferred response) triples train a reward model the same way RLHF would, but the labels are AI-generated. The SL-CAI model is then RL-tuned against this AI-derived reward model.
Training time only
The constitution does not run at inference. By the end of training, the principles are baked into the model weights. At inference, there is no document being consulted. The model behaves as if the constitution were second nature. This is sometimes confused with a runtime system prompt; it is not.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's post 'Claude's Constitution' (May 2023) says the constitution is used in two places in Claude's training, the supervised phase and the RL phase; Anthropic published a new version of the constitution in January 2026.
- Lee et al. (Google, 2023) found that RLAIF reaches performance comparable to RLHF on summarization, helpful dialogue and harmless dialogue.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you stress-test a constitution before shipping a model trained on it?
Generate adversarial prompts that probe the principles for unstated edge cases, then run a small RLHF-trained baseline against the CAI-trained model on the same prompts. Constitution gaps show up as the CAI model deviating consistently from human judgments on one principle's scope.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling Constitutional AI 'just RLHF with an AI labeller'; it is also a different supervised stage (critique and revise) before the preference model is even trained.
60 second bullets to scan on the way to the call.
Where the policy lives in RLHF versus Constitutional AI
The two training stages of CAI (SL-CAI self-critique, RL-CAI preference)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.