Constitutional AI replaces most human preference labels with model self-critique against a written principles list,the policy becomes editable text instead of implicit labelling.
Imagine teaching someone manners. RLHF is like having a tutor sit with the student and grade every conversation: harsh, slow, and the tutor's taste leaks into every grade. Constitutional AI is like giving the student a small written rulebook and asking them to grade their own conversations against it, then revise. A second judge model checks the revisions. The rulebook is short and human-readable, so when the rules change, you do not need to retrain the tutor,you edit a page. The student still practices and learns the same skill of being helpful and harmless; the change is in who holds the rubric and how visible it is.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Constitutional AI is the most-cited example of what changes when you replace human preference labels with model-generated ones, but the popular framing,'CAI is RLHF with AI labels',misses the more interesting structural difference. The shift is not just who labels the preferences. It is also where the safety policy lives, how it is reviewed, and how quickly it can change.
This deep dive walks through the two-stage training loop, the governance properties an explicit constitution unlocks, the cost and consistency story behind AI feedback, and the failure modes that explain why frontier labs in 2026 converge on hybrid pipelines rather than pure CAI or pure RLHF.
Mental model: RLHF stores its safety policy as a billion preference labels held implicitly in a reward model's weights. Constitutional AI stores it as a few pages of text plus the critic model that knows how to apply them. One you audit by inspecting outputs; the other you audit by reading the document.
The two-stage training loop in detail
Stage one: SL-CAI (supervised learning, constitutional)
The model is prompted with a potentially harmful or ambiguous question. It generates an initial response. Then a second prompt feeds the model its own response along with a randomly sampled principle from the constitution, and asks the model to critique whether the response violates that principle. A third prompt asks the model to rewrite the response addressing the critique.
The (original prompt, final rewritten response) pair becomes a supervised fine-tuning example. After thousands of such pairs across the prompt distribution, the base model is fine-tuned on this dataset. The model now has internalised the revision behaviour: when it sees a similar prompt, the revised-style answer is closer in the loss landscape.
Stage two: RL-CAI / RLAIF (reinforcement learning from AI feedback)
A second model,usually the SL-CAI model from stage one, or a separate judge model,is given two candidate responses to the same prompt and asked to pick the one that better satisfies the constitution. This is run at scale: hundreds of thousands of prompts, many candidate pairs each.
The (prompt, preferred response, dispreferred response) triples train a reward model the same way RLHF would, but the labels are AI-generated. The base model is then RL-tuned (PPO or DPO) against this AI-derived reward model.
The training time only nature
The constitution does not run at inference. By the end of training, the principles are baked into the model weights. At inference, there is no document being consulted,the model behaves as if the constitution were second nature. This is sometimes confused with a runtime system prompt; it is not.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude family (Claude 3.5, Sonnet 4.6, Opus 4.7) is trained with Constitutional AI as the dominant feedback signal; the published constitution evolves across model generations.
- Google's Gemini 3.1 incorporates RLAIF for safety policies at training scale, with human review concentrated on the highest-impact categories.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you stress-test a constitution before shipping a model trained on it?
Generate adversarial prompts that probe the principles for unstated edge cases, then run a small RLHF-trained baseline against the CAI-trained model on the same prompts. Constitution gaps show up as the CAI model deviating consistently from human judgments on one principle's scope.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling Constitutional AI 'just RLHF with an AI labeller',it is also a different supervised stage (critique and revise) before the preference model is even trained.
60 second bullets to scan on the way to the call.
Where the policy lives in RLHF versus Constitutional AI
The two training stages of CAI (SL-CAI self-critique, RL-CAI preference)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.