Explain Constitutional AI and how it reduces reliance on human preference labels
Explain Constitutional AI and how it reduces reliance on human preference labels.
Constitutional AI replaces human harmlessness labels with model labels: a model critiques and revises its answers against written principles, then an AI judge picks the less harmful answer to train the reward.
Think of a cooking school with one head chef and hundreds of students. The chef cannot taste every dish, so the chef writes a short card of rules: no raw chicken, nothing burnt, nothing too salty. First, each student cooks a dish, checks it against one rule from the card, and fixes it. The school keeps the fixed dishes as examples to learn from. Next, a senior student tastes two dishes and, using one rule from the card, says which is safer to serve. Those picks train a scorer, and the students practice until the scorer likes their food. The head chef still writes the card and still judges whether the food tastes good. Constitutional AI works the same way: the rules replace most of the human safety tasting, not all human input.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Constitutional AI (CAI) is a training method from Anthropic's 2022 paper "Constitutional AI: Harmlessness from AI Feedback". It trains an assistant to be harmless without any human labels that mark outputs as harmful. Human input for harmlessness shrinks to a short list of natural-language principles, the constitution, plus a few hand-written examples for few-shot prompting.
The method keeps the shape of RLHF but changes who does the labor-heavy steps. In a supervised stage, a model rewrites its own harmful answers. In an RL stage, a model picks the more harmless of two answers. Both steps read one principle at a time from the constitution.
What the constitution replaces, and what it does not
RLHF for a helpful and harmless assistant needs two kinds of human comparisons. Crowdworkers pick the more helpful of two answers, and they pick the less harmful of two answers to red-team prompts. Anthropic's earlier HH RLHF work used human labels for both. RLHF already rewards the policy with a learned preference model, but that model is trained on a large set of human labels.
CAI removes the second kind. In the RL stage, a model labels every harmlessness comparison, while helpfulness comparisons stay human. The preference model is a hybrid: 135,296 human helpfulness comparisons and 182,831 AI-generated harmlessness comparisons. The paper notes it could have mixed human and AI labels for both, but kept harmlessness fully AI-labeled to show that the technique works.
The constitution itself is small. The paper wrote 16 principles for the supervised stage and 16 for the RL stage, chosen in an ad hoc and iterative way for research. A typical RL principle reads: "Which of these assistant responses is less harmful? Choose the response that a wise, ethical, polite and friendly person would more likely say." Many principles overlap, and some target specific harms. Humans also write the few-shot examples that show the critique, revision, and labeling formats.
Reducing reliance on human labels therefore has a precise meaning. Human effort moves from labeling hundreds of thousands of harmlessness pairs to writing a few dozen rules and a handful of demonstrations. It does not mean the pipeline runs with no human input.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Constitutional AI paper trained 52B models with 182,831 AI-labeled harmlessness comparisons and no human harmlessness labels.
- Anthropic's May 2023 post on Claude's constitution lists principle sources such as the UN Universal Declaration of Human Rights, Apple's terms of service, and the Sparrow rules.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy were the RL answer pairs generated by SL-CAI and not by the original helpful RLHF model?
Think about how close the preference model's training answers are to what the policy samples early in RL.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying Constitutional AI needs no humans. Humans write the principles and examples, and in the original paper they still labeled helpfulness; only the harmlessness labels came from AI.
60 second bullets to scan on the way to the call.
What the constitution contains and who writes it
Which human labels CAI removes and which it keeps
Primary sources. Browse if you want the original framing.
- Bai et al., Constitutional AI: Harmlessness from AI Feedback
- Lee et al., RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- Anthropic, Claude's constitution
- Huang et al., Collective Constitutional AI: Aligning a Language Model with Public Input
- Hugging Face, Constitutional AI with Open LLMs
Same topic, related formats. Practice these next.