Order the stages of a constitutional RLAIF pipeline

Order steps·Medium·4.0 · 0·~1 min·Asked atCitadel Databricks Moveworks

Attempt it

1The AI revises responses to remove the critiqued violations
2Sample initial responses from a helpful only model
3Generate response pairs and have an AI labeler pick winners using the constitution
4Run RL against that reward model (the RLAIF step)
5SFT the model on the revised responses
6Train a preference (reward) model on the AI generated labels
7An AI critic flags where each response violates the constitution's principles

Constitutional RLAIF (the Anthropic Constitutional AI recipe) has two clear stages. **Stage 1: SL-CAI (Supervised Learning from Constitutional AI).** 1. Sample initial responses from a helpful-only model on a range of prompts, including red-team prompts designed to elicit problems. 2. An AI critic reads each response and lists where it violates principles from the constitution. 3. The same model rewrites the response to remove the flagged violations. 4. Fine-tune the model on the revised responses with standard supervised learning. The model now produces safer outputs by default. **Stage 2: RL-CAI (the RLAIF step).** 5. Generate response pairs for many prompts. An AI labeller compares each pair against the constitution and picks the better one. 6. Train a reward model on those AI-generated preference labels. 7. Run RL (PPO, or increasingly DPO) against the reward model. The order matters. The supervised stage gives the RL stage a safer starting point so the policy is not blowing up the KL anchor right out of the gate. The critique then revise loop produces the supervised data that no human labeller produced. The preference loop is the RLAIF step proper, where AI feedback fully replaces human preference labels.

**Constitutional RLAIF** is a two-stage pipeline. The first stage is supervised (SL-CAI), the second is reinforcement (RL-CAI). The order matters because each stage prepares the input the next stage needs. **Stage 1: SL-CAI, critique revise finetune.** A helpful-only model produces initial responses to a broad prompt distribution, including red-team prompts. An AI critic, working from the constitution, lists violations in each response. The model rewrites the response addressing the violations. The pairs (original prompt, revised response) become a supervised fine-tuning dataset. The model is fine-tuned on this dataset. The output is a model that already produces constitution-compliant responses on most prompts. **Stage 2: RL-CAI, the RLAIF step.** With the SL-CAI checkpoint as the new starting policy, generate pairs of responses for each prompt. An AI labeller (a constitution-following judge) compares each pair and outputs a preference. Those preferences train a reward model. The policy is then RL-tuned against the reward model, typically with PPO or, in 2026 pipelines, increasingly DPO. **Why this order.** Going straight to RL from a helpful-only base means PPO's KL anchor is fighting a policy still producing unsafe outputs by default; the gradient and the anchor pull in different directions. SL-CAI moves the base policy into the constitution-compliant region first, so the RL stage only needs to refine, not transform. Also, SL-CAI produces the critique-revise data that no human ever labelled, and the AI labeller produces the preference labels that no human ever labelled. Together they make a fully AI-feedback pipeline tractable.

TL;DR

Constitutional RLAIF runs two stages: an SL-CAI critique revise finetune stage, then an RL-CAI stage where an AI labeller produces preferences against the constitution and PPO or DPO uses the resulting reward model.

Memory aid

Easy to grasp

Think of training a new editor at a magazine that has a written style guide. First you take their early drafts, mark every place the draft breaks a rule, and have them rewrite. Once they have rewritten a few thousand drafts that way, you train them by feeding the corrected versions back as examples (the supervised stage). Then you give them pairs of articles and ask which one follows the style guide better. Their answers train a grader, and the editor practices against the grader until their natural draft already passes (the reinforcement stage). Constitutional RLAIF does the same thing for safety with the constitution as the style guide.

Key concepts

RLHF & Alignment

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Constitutional RLAIF is the full Anthropic Constitutional AI recipe, and it is the production answer to the question 'how do we train a frontier safety model when human preference labelling cannot keep up?' The pipeline has seven concrete steps that live in two stages: a supervised stage that uses an AI critic to produce safe training data, and a reinforcement stage that uses an AI labeller to produce preference data.

Missing the structure of the pipeline is the most common mistake in interviews on this topic. Candidates often collapse the two stages into one, or place preference labelling before the critique-revise SFT, or forget that the helpful-only starting point is intentional rather than a bug.

Mental model: SL-CAI moves the base model into the constitution-compliant region; RL-CAI sharpens the boundary. Both are necessary, in that order, because each stage prepares the input the next stage needs.

Stage 1: SL-CAI, supervised learning from critique and revise

The starting point. A helpful-only model is the right base. It must be fluent and willing to attempt every prompt, including the red-team ones, because critique-revise needs violations to remove. A base model already RLHF'd for harmlessness will refuse the red-team prompts and produce no critique-revise data.

Step 1: sample initial responses. Choose a prompt distribution that includes ordinary user requests plus a heavy red-team component. The red-team prompts surface the violations the constitution exists to address. Sampling temperature is typically modest to keep responses on-distribution.

Step 2: AI critic flags violations. The critic is a constitution-following model. It reads the prompt, the response, and the constitution, and produces a structured list of which principles are violated and where. The critic can be the same model wearing a different prompt, or a different model trained for the role.

Step 3: AI revises. The same (or a sibling) model rewrites the response with instructions to address the flagged violations. In some recipes the critique-revise loop is iterated two or three times to converge to a clean revision.

Step 4: SFT on revised responses. The (prompt, revised-response) pairs become a supervised dataset. The base model is fine-tuned on it. The resulting checkpoint already produces constitution-compliant responses on most prompts; it has internalised the policy as a default behaviour, not just as a wrapper.

What this stage produces. A model that no longer needs explicit instructions to be safe on the categories the constitution covers, plus a corpus of (violation, revision) pairs that is itself valuable as evaluation data and as input to constitution iteration.

Stage 2: RL-CAI, the RLAIF step proper

Why the order is locked

Where this lives in 2026 production

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic Claude Opus 4.7 and Claude Sonnet 4.6 are trained with constitutional RLAIF as a major component; the constitution is published and evolves across model generations.
DeepSeek-R1 uses GRPO with verifiable and constitutional-style rewards combined; the constitutional RLAIF pattern shows up in its safety post-training stage.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhere does DPO fit in this pipeline and how does it change the order?

DPO replaces the reward model then-RL substep in stage 2 with a direct preference loss. The seven steps collapse to six: skip the reward model and train the policy directly from the AI-labelled pairs. The SL-CAI stage is unchanged.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Skipping the SL-CAI critique-revise stage and going straight from preference labels to RL; the supervised stage is what gives the policy a safe starting point for RL.

60 second bullets to scan on the way to the call.

The two-stage split: SL-CAI then RL-CAI
The critique-revise loop and what it produces

Primary sources. Browse if you want the original framing.