Judge a strict easy to hard curriculum against mixture plus annealing

MCQ·Medium·4.0 · 0·~1 min·Asked atAnthropic Google Meta

Attempt it

Two extreme positions are both wrong in this debate. **Strict easy to hard curriculum** sounds intuitive, the way you might teach a human student. But at LLM scale, where the model sees trillions of tokens shuffled from a wide mixture, published evidence for sorting the entire corpus by difficulty is weak and inconsistent. The wins, when they appear, are typically small and not worth the pipeline complexity of defining 'difficulty' for 10T tokens of mixed content. **Data order is fully irrelevant** is also wrong, and it is the more dangerous mistake. The end of training is special. The learning rate has decayed close to zero, the model has stopped exploring, and the last gradient updates are what stick most strongly in the final weights. This is the **annealing phase**. What reliably pays off is two things working together: a well-tuned mixture sampled shuffled throughout the bulk of training, and a deliberate concentration of high-quality data in the final 5 to 20 percent of training where annealing happens. That late-phase quality boost is now standard practice at frontier labs (Llama 3, DeepSeek, Qwen all describe variants of it).

The defensible position separates two regimes: the bulk of pretraining and the annealing tail. - **Bulk training.** Tokens are shuffled from a tuned mixture across web, code, math, multilingual, and high-quality sources. Strict global difficulty ordering at this scale has shown small, inconsistent wins in the literature and adds significant pipeline complexity to define and maintain. The default of shuffled mixture sampling remains correct. - **Annealing tail.** The last 5 to 20 percent of training, when the LR cosine or WSD schedule is decaying toward its minimum, behaves differently. The optimizer has stopped exploring and the model's final weights are largely determined by what it sees in this window. Frontier teams concentrate high-quality data (curated math, code, instruction-style text, deduplicated technical content) here and report reliable gains on downstream evals. Why the asymmetry exists: high LR updates are noisy and largely overwritten by subsequent steps; low LR updates compound directly into the final weights. So order matters in a very specific way: it matters most for the data shown when the LR is smallest. The correct answer to the PM is therefore to drop the corpus-wide difficulty sort and instead invest in (1) better mixture tuning and (2) a deliberate late-phase data curation pipeline that feeds the annealing window. This is what most frontier-scale recipes published since 2024 actually do.

Quick reference

TL;DR

Strict global curricula barely pay off at LLM scale, but the LR decay phase is genuinely special, which is why labs concentrate high-quality data into the late annealing window.

Memory aid

Easy to grasp

Picture studying for a year-long exam. Spending the first ten months sorting your textbooks from easiest to hardest, then reading them in order, makes very little difference at the end. The textbooks you read in the final two weeks, when you are calm and your memory is freshest, do matter. Pretraining works the same way. Sorting the whole corpus by difficulty is mostly wasted effort, but the data placed in the last few percent of training, during the LR decay phase, is the data the model retains most strongly.

Key concepts

Pretraining

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 minutes: why strict global curricula underperform at scale, why the annealing phase is mechanically different, the LR-weighted view that explains the asymmetry, and the late-phase quality-concentration recipe used by frontier labs.

Real-world examples

Real products, models, and research that use this idea.

Llama 3 explicitly describes a high-quality annealing data mixture for the final phase of pretraining, with measurably better downstream eval results than no-annealing controls
DeepSeek-V3's two-stage schedule reserves a deliberately curated math and code heavy mixture for the WSD decay phase rather than sorting the full corpus by difficulty
Qwen 2.5 documentation reports a multi-stage training schedule with quality-weighted upsampling in the final phase, not a corpus-wide curriculum sort
OLMo 2's mid-training and annealing experiments publish per-phase mixture weights and confirm the late-phase data quality has outsized impact on retained scores

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you decide what data goes into the annealing window?

QWhat is the difference between WSD annealing and cosine annealing for the late phase?

QWhy does sorting the corpus by document length fail as a curriculum signal?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Picking between 'order matters' and 'order does not matter' as a binary, instead of recognizing the asymmetry between the bulk of training (order weakly matters) and the annealing phase (order strongly matters).

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.