1Split into train / dev / holdout, store as {prompt, chosen, rejected} triples, and freeze the seed used for candidate generation so the dataset is reproducible.
2Have humans (or a strong judge model) rank or pairwise-compare the candidates per prompt, producing a chosen/rejected label for each pair.
3Filter pairs where the quality gap is too small to learn from (e.g. judge confidence below a threshold, or rubric tie), keeping only pairs with a clear preferred side.
4Deduplicate near-identical chosen and rejected texts (and drop pairs where chosen is approximately rejected) so the model is not asked to learn distinctions between essentially the same answer.
5For each prompt, generate 4-8 candidate responses from the current SFT model at temperature ~1.0 to get genuine diversity, not near-identical samples.
6Sample ~5,000 prompts from the production traffic distribution (or a proxy) so the trained preference signal matches what users actually ask.
**The correct order is 0, 1, 2, 3, 4, 5.** Each step needs the output of the previous step to exist, so the sequence is largely forced.
First, you need prompts. These come from production traffic or a proxy that matches what users actually ask. Picking prompts later would mean generating candidates before knowing what to generate for.
Second, you generate candidate responses for each prompt. Multiple candidates per prompt at a non-zero temperature give you real diversity to choose between.
Third, you label the candidates. Humans or a strong judge model rank them, producing chosen and rejected pairs. The labelling step needs candidates to exist first.
Fourth, you deduplicate. Many candidates from the same model are near-identical; pairs where chosen and rejected are essentially the same text teach the model nothing useful.
Fifth, you filter for quality gap. Pairs where the chosen response is only marginally better than the rejected one carry low signal; keeping them dilutes the training loss with near-random labels.
Sixth, you split and store. Train, dev, and holdout splits go to the trainer in the standard format, with the random seed frozen for reproducibility.
The two filtering passes specifically belong between labelling and splitting because both filters need the labels to operate.
**The order is 0, 1, 2, 3, 4, 5.** The sequence is dictated by data dependencies: each step needs the artifact produced by the previous step.
Step 0 sources prompts from real traffic or a representative proxy. This anchors the preference signal to what users actually ask, not to academic prompt distributions.
Step 1 generates 4 to 8 candidates per prompt from the current SFT model at temperature around 1.0. Multiple candidates and a non-trivial sampling temperature are both load-bearing; without diversity you get pairs that are too similar to learn from.
Step 2 labels the candidates. Humans or a strong judge model produce chosen and rejected pairs. The labelling needs candidates to exist first, and the size of the labelling budget often determines how many prompts you can include.
Steps 3 and 4 are the two filtering passes, both placed after labelling because both need the labels. Step 3 drops near-identical text pairs, which arise when the model produces close variants of the same answer. Step 4 drops low-confidence pairs where the chosen and rejected responses are essentially tied; these pairs carry near-random labels and harm training.
Step 5 splits the cleaned data into train, dev, and holdout sets, stores them in the canonical {prompt, chosen, rejected} format, and freezes the generation seed so the dataset can be reproduced or extended consistently.
The filtering passes are placed between labelling and splitting because earlier would be impossible (no labels yet) and later would corrupt the splits with noise.
Quick reference
TL;DR
Prompts first, then candidates, then labels, then two filtering passes (deduplication and quality-gap), then the final split. Each step depends on the artifact the previous step produced.
Think of judging a baking competition. You first invite contestants (the prompts), then collect their cakes (the candidate responses), then have judges score and rank them (the labels). Before you publish the results, you check that no two cakes are identical entries and that the winning cake is meaningfully better than the runner-up. Only then do you split the results into the official records and an audit set. Doing these steps out of order means judges score nothing, or you keep ties as wins, or you train your taste critic on noise. The sequence reflects what each step needs to exist first.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You are doing online DPO where pairs are generated and labelled inside the training loop The offline pipeline collapses into the training step; filtering happens on the fly, and the splitting concept does not apply in the same way.
You already have a high-quality open preference dataset that matches your distribution Datasets like UltraFeedback can be reused directly without rebuilding the pipeline; the curation steps only apply when you are creating fresh data.
You are doing KTO or RLOO that does not require paired data Those methods consume unpaired signal (binary good or bad labels for KTO, group rollouts for RLOO), so the pair-construction steps do not apply.
Quick scan · 2–4 min
Everything important, quickly.
3 min: walk through the six-step dependency chain, emphasise that the two filter passes go between labelling and splitting, and explain why each filter is load-bearing.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's published RLHF data pipelines follow this same dependency order, with prompts sourced from production traces and candidates generated from successive policy versions.
2UltraFeedback, the most-cited open DPO dataset in 2026, uses a strong judge model for labelling and explicitly documents its near-duplicate and quality-gap filtering rules.
3Hugging Face TRL's DPO documentation expects the data in {prompt, chosen, rejected} format produced exactly by this kind of pipeline, regardless of whether labelling was human or model-based.
4Allen AI's Tulu 3 release used a multi-source preference dataset assembled following this six-step pattern, with per-stage ablations published.
5Production teams fine-tuning Llama 4 Maverick or DeepSeek V4 with DPO commonly use Claude Opus 4.7 or GPT-5.5 as the judge model, with random human spot-checks to validate label quality.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide between a strong judge model and human labellers for the labelling step, and how would you combine them?
Sign in or go Pro to see the approach for this follow-up.
QWhy is the temperature 1.0 choice for candidate generation important, and what changes if you use temperature 0.7 or 1.5?
Sign in or go Pro to see the approach for this follow-up.
QWhat would happen if you skipped the quality-gap filter and trained DPO on all labelled pairs including ties?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Skipping the quality-gap filter and training on labelled pairs where the chosen and rejected responses are essentially tied. That teaches the model to chase noise rather than real preference signal.