Order a 2026 post-training pipeline from base checkpoint to deployed assistant

Order steps·Easy·4.0 · 0·~1 min·Asked atHaptik Ironclad Replicate

Attempt it

1Collect preference labels on sampled response pairs
2Pretrain the base model on web scale text
3Run RL (PPO or GRPO) against the reward model with a KL anchor
4Deploy, then collect live feedback to seed the next round
5Train a reward model on the preference data
6Red team and run safety plus capability evals on the candidate
7SFT on curated demonstrations to get an instruct checkpoint

**The order is fixed by data dependencies.** Each stage produces what the next stage needs. First, **pretraining** on web-scale text gives a base model with broad linguistic competence but no instruction-following. Second, **SFT on curated demonstrations** turns that base into an instruct checkpoint that can answer questions in a useful format. Third, **preference label collection** samples response pairs from the SFT model (and sometimes from earlier checkpoints) and asks annotators which is better. You cannot do this before SFT because the base model produces too little useful output to label. Fourth, **reward-model training** uses the preference pairs to fit a scalar quality score via Bradley-Terry loss. Fifth, **RL (PPO or GRPO)** runs the policy against the reward model with a KL anchor against the SFT model to prevent drift. Sixth, **red-team and evals** gate the deploy decision: safety, capability, refusal calibration, regression checks. Seventh, **deploy and collect live feedback**, which seeds the next round of preference labels and closes the loop.

**Canonical order:** Pretrain to SFT to preference collection to reward-model training to RL to evals to deploy, with deploy feeding the next round. **Why this order is forced by dependencies.** - Pretraining produces the language-modeling competence; without it nothing downstream has a foundation. - SFT on demonstrations gives the model basic instruction-following. Preference collection needs a model that can produce plausible candidate responses; the raw base model is not useful for that. - Preference collection turns SFT samples into chosen/rejected pairs annotated by humans (or by a calibrated AI labeler with human audits). - Reward-model training fits Bradley-Terry on those pairs to get a scalar reward function. - RL (PPO or GRPO) optimizes the policy against the RM, regularized by KL against the SFT model so the policy does not drift into incoherence. - Red-team and evals (safety, capability, refusal calibration) are the gate before any rollout. - Deployment collects live feedback that seeds the next iteration's preference data. **Common variants.** DPO replaces explicit RM training and RL with a single offline preference-optimization step, but the surrounding order is unchanged. Constitutional RLAIF replaces some of the human preference labels with model-based critique against a written constitution. GRPO replaces PPO's per-sample advantage with group-relative advantages. The order of operations stays the same; only what runs at the RL stage shifts.

TL;DR

Pretrain, SFT, preference labels, reward model, RL, evals, deploy. Each stage's output is the next stage's input, and deployment seeds the next round.

Memory aid

Easy to grasp

Think about training a new employee. First they go to school and learn general things (pretraining). Then they shadow experienced staff and copy how those people handle real cases (SFT on demonstrations). Then their manager watches them work on pairs of similar tasks and writes down which approach was better (preference labels). The manager turns those notes into a scoring rubric (the reward model). The employee practices and gets coached using the rubric (RL). Before going customer-facing, they pass a checklist of evaluations (red-teaming, safety, capability). Once on the floor, real customer feedback flows back into the next round of coaching. Each step depends on the one before it.

Key concepts

RLHF

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The post-training pipeline is the most important sequence in modern LLM engineering, and it is also one of the most-frequently scrambled in interviews. The order is not arbitrary: each stage produces a specific artifact that the next stage requires. Reorder it and the downstream stages have no valid input.

This deep dive walks through the seven canonical stages, explains why each dependency is strict, names the modern variants that change runtime but preserve the order, and discusses how the deploy to next round feedback loop actually closes in 2026 production practice.

The pedagogical goal is to leave you able to name the order from memory, explain why a candidate's alternative ordering does not satisfy the data-flow constraints, and place techniques like DPO, GRPO, RLVR, and constitutional RLAIF inside the canonical structure rather than treating them as parallel pipelines.

Stages 1-3: pretrain, SFT, preference collection

Pretraining produces the language model. A base model trained on web-scale text learns the joint distribution over tokens; it can complete sentences and continue passages but cannot follow instructions in a useful format. Pretraining is the foundation everything else builds on. It is also computationally dominant; the rest of the pipeline is a thin layer on top.

SFT (supervised fine-tuning on demonstrations) takes the base model and fine-tunes it on curated examples of instruction following. The result is an instruct checkpoint that responds to prompts in a useful format. SFT data is expensive per token but small in volume compared to pretraining. The output of SFT is the policy that subsequent stages will sample from.

Preference collection is the first stage that requires the policy to be useful. Annotators see prompts and two candidate responses (sampled from the SFT model at varying temperatures, or from current and previous checkpoints), and pick the better one. Annotating base-model samples here wastes budget because the candidates are too incoherent to give useful signal. The output is a corpus of (prompt, chosen, rejected) tuples.

Stages 1-3 in concrete terms. Pretraining produces a base model with next-token prediction on a multi trillion token corpus (Llama 4 mid-2025 used roughly 15T tokens; Qwen 3 around 18T; DeepSeek-V3 about 14T). SFT (stage 2) does instruction tuning on a curated dataset of perhaps 50K to 5M instruction-response pairs, depending on team. Stage 3 is preference data collection: pairwise comparisons from human annotators or LLM judges, scaled to 100K to 1.5M pairs for frontier models. The Llama 3 paper reports 1.5M preference pairs across its iterative pipeline. Anthropic's 2024 system cards report similar scales.

Stages 4-5: reward model and RL

Stages 6-7: evals and deploy to feedback

Modern variants and how they fit

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI InstructGPT and the broader GPT-5.5 lineage follow this canonical order with PPO at the RL stage
Anthropic's Claude Opus 4.7 uses constitutional RLAIF at the preference and RL stages while keeping the surrounding order intact

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does DPO restructure the pipeline without changing the order?

DPO derives the policy update directly from preference pairs against the SFT reference, eliminating the explicit RM and the PPO loop. The same data flows but two stages merge.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Swapping the order of SFT and preference collection, or collecting preferences before there is any policy worth sampling from. The dependency graph forces this order.

60 second bullets to scan on the way to the call.

Why each stage's output is required by the next stage
What changes when DPO replaces classical RM-plus-PPO

Primary sources. Browse if you want the original framing.