Two extreme positions are both wrong in this debate.
**Strict easy to hard curriculum** sounds intuitive, the way you might teach a human student. But at LLM scale, where the model sees trillions of tokens shuffled from a wide mixture, published evidence for sorting the entire corpus by difficulty is weak and inconsistent. The wins, when they appear, are typically small and not worth the pipeline complexity of defining 'difficulty' for 10T tokens of mixed content.
**Data order is fully irrelevant** is also wrong, and it is the more dangerous mistake. The end of training is special. The learning rate has decayed close to zero, the model has stopped exploring, and the last gradient updates are what stick most strongly in the final weights. This is the **annealing phase**.
What reliably pays off is two things working together: a well-tuned mixture sampled shuffled throughout the bulk of training, and a deliberate concentration of high-quality data in the final 5 to 20 percent of training where annealing happens. That late-phase quality boost is now standard practice at frontier labs (Llama 3, DeepSeek, Qwen all describe variants of it).
The defensible position separates two regimes: the bulk of pretraining and the annealing tail.
- **Bulk training.** Tokens are shuffled from a tuned mixture across web, code, math, multilingual, and high-quality sources. Strict global difficulty ordering at this scale has shown small, inconsistent wins in the literature and adds significant pipeline complexity to define and maintain. The default of shuffled mixture sampling remains correct.
- **Annealing tail.** The last 5 to 20 percent of training, when the LR cosine or WSD schedule is decaying toward its minimum, behaves differently. The optimizer has stopped exploring and the model's final weights are largely determined by what it sees in this window. Frontier teams concentrate high-quality data (curated math, code, instruction-style text, deduplicated technical content) here and report reliable gains on downstream evals.
Why the asymmetry exists: high LR updates are noisy and largely overwritten by subsequent steps; low LR updates compound directly into the final weights. So order matters in a very specific way: it matters most for the data shown when the LR is smallest.
The correct answer to the PM is therefore to drop the corpus-wide difficulty sort and instead invest in (1) better mixture tuning and (2) a deliberate late-phase data curation pipeline that feeds the annealing window. This is what most frontier-scale recipes published since 2024 actually do.
Quick reference
TL;DR
Strict global curricula barely pay off at LLM scale, but the LR decay phase is genuinely special, which is why labs concentrate high-quality data into the late annealing window.
Picture studying for a year-long exam. Spending the first ten months sorting your textbooks from easiest to hardest, then reading them in order, makes very little difference at the end. The textbooks you read in the final two weeks, when you are calm and your memory is freshest, do matter. Pretraining works the same way. Sorting the whole corpus by difficulty is mostly wasted effort, but the data placed in the last few percent of training, during the LR decay phase, is the data the model retains most strongly.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Small-scale or short-budget runs where the model never reaches a long enough horizon for annealing to matter The bulk vs annealing asymmetry depends on a long-enough training schedule; very short runs collapse the distinction
Domain-specific fine-tuning where the data is narrow and uniformly relevant Curriculum considerations apply less strongly when the data is not a diverse mixture; sorting by some difficulty proxy may help in narrow settings
Highly constrained-vocabulary training (code-only, math-only) where difficulty is measurable and consistent Constrained corpora may admit principled difficulty signals that mixed-web corpora do not
Quick scan · 2–4 min
Everything important, quickly.
6 minutes: why strict global curricula underperform at scale, why the annealing phase is mechanically different, the LR-weighted view that explains the asymmetry, and the late-phase quality-concentration recipe used by frontier labs.
Real-world examples
Real products, models, and research that use this idea.
1Llama 3 explicitly describes a high-quality annealing data mixture for the final phase of pretraining, with measurably better downstream eval results than no-annealing controls
2DeepSeek-V3's two-stage schedule reserves a deliberately curated math and code heavy mixture for the WSD decay phase rather than sorting the full corpus by difficulty
3Qwen 2.5 documentation reports a multi-stage training schedule with quality-weighted upsampling in the final phase, not a corpus-wide curriculum sort
4OLMo 2's mid-training and annealing experiments publish per-phase mixture weights and confirm the late-phase data quality has outsized impact on retained scores
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide what data goes into the annealing window?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the difference between WSD annealing and cosine annealing for the late phase?
Sign in or go Pro to see the approach for this follow-up.
QWhy does sorting the corpus by document length fail as a curriculum signal?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Picking between 'order matters' and 'order does not matter' as a binary, instead of recognizing the asymmetry between the bulk of training (order weakly matters) and the annealing phase (order strongly matters).