Explain how Constitutional AI differs from vanilla RLHF

Attempt it

RLHF learns from human preference rankings,humans choose which of two model outputs is better. Constitutional AI replaces most of that human labelling with model self-critique against a written list of principles: the model generates an output, critiques it against the constitution, rewrites it, and the revised version is used for preference training. The constitution is explicit text, so policy changes do not require a new labelling campaign,they require an edit to the principles.

Your answer

Quick reference

TL;DR

Constitutional AI replaces most human preference labels with model self-critique against a written principles list,the policy becomes editable text instead of implicit labelling.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine teaching someone manners. RLHF is like having a tutor sit with the student and grade every conversation: harsh, slow, and the tutor's taste leaks into every grade. Constitutional AI is like giving the student a small written rulebook and asking them to grade their own conversations against it, then revise. A second judge model checks the revisions. The rulebook is short and human-readable, so when the rules change, you do not need to retrain the tutor,you edit a page. The student still practices and learns the same skill of being helpful and harmless; the change is in who holds the rubric and how visible it is.

Key concepts

RLHF Safety & Guardrails

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

7 min: where the policy lives in each method, the two CAI training stages (SL-CAI plus RL-CAI), why a written constitution wins on audit, scale and consistency advantages of AI feedback, failure modes (misspecification, weak critic), why hybrid wins in 2026.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's Claude family (Claude 3.5, Sonnet 4.6, Opus 4.7) is trained with Constitutional AI as the dominant feedback signal; the published constitution evolves across model generations.
Google's Gemini 3.1 incorporates RLAIF for safety policies at training scale, with human review concentrated on the highest-impact categories.
Open-source projects like RLAIF-V and Constitutional DPO reimplement the loop on open-weights models such as Llama 4 and Qwen 3.5.
Safety-evaluation suites like TrustLLM and MLCommons AILuminate benchmark CAI-trained and RLHF-trained models on identical adversarial prompts to expose policy differences.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you stress-test a constitution before shipping a model trained on it?

Sign in or go Pro to see the approach for this follow-up.

QWhat goes wrong when the critic model is significantly weaker than the model being trained?

Sign in or go Pro to see the approach for this follow-up.

QWhen is RLAIF without the supervised critique stage worth it versus the full CAI pipeline?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Calling Constitutional AI 'just RLHF with an AI labeller',it is also a different supervised stage (critique and revise) before the preference model is even trained.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.