Judge a strict easy to hard curriculum against mixture plus annealing

MCQ·Medium·4.0 · 0·~1 min·Asked atInfosys Robust Intelligence Zepto

Attempt it

Two extreme positions are both wrong in this debate. **Strict easy to hard curriculum** sounds intuitive, the way you might teach a human student. But at LLM scale, where the model sees trillions of tokens shuffled from a wide mixture, published evidence for sorting the entire corpus by difficulty is weak and inconsistent. The wins, when they appear, are typically small and not worth the pipeline complexity of defining 'difficulty' for 10T tokens of mixed content. **Data order is fully irrelevant** is also wrong, and it is the more dangerous mistake. The end of training is special. The learning rate has decayed close to zero, the model has stopped exploring, and the last gradient updates are what stick most strongly in the final weights. This is the **annealing phase**. What reliably pays off is two things working together: a well-tuned mixture sampled shuffled throughout the bulk of training, and a deliberate concentration of high-quality data in the final 5 to 20 percent of training where annealing happens. That late-phase quality boost is now standard practice at frontier labs (Llama 3, DeepSeek, Qwen all describe variants of it).

The defensible position separates two regimes: the bulk of pretraining and the annealing tail. - **Bulk training.** Tokens are shuffled from a tuned mixture across web, code, math, multilingual, and high-quality sources. Strict global difficulty ordering at this scale has shown small, inconsistent wins in the literature and adds significant pipeline complexity to define and maintain. The default of shuffled mixture sampling remains correct. - **Annealing tail.** The last 5 to 20 percent of training, when the LR cosine or WSD schedule is decaying toward its minimum, behaves differently. The optimizer has stopped exploring and the model's final weights are largely determined by what it sees in this window. Frontier teams concentrate high-quality data (curated math, code, instruction-style text, deduplicated technical content) here and report reliable gains on downstream evals. Why the asymmetry exists: high LR updates are noisy and largely overwritten by subsequent steps; low LR updates compound directly into the final weights. So order matters in a very specific way: it matters most for the data shown when the LR is smallest. The correct answer to the PM is therefore to drop the corpus-wide difficulty sort and instead invest in (1) better mixture tuning and (2) a deliberate late-phase data curation pipeline that feeds the annealing window. This is what most frontier-scale recipes published since 2024 actually do.

TL;DR

Strict global curricula barely pay off at LLM scale, but the LR decay phase is genuinely special, which is why labs concentrate high-quality data into the late annealing window.

Memory aid

Easy to grasp

Picture studying for a year-long exam. Spending the first ten months sorting your textbooks from easiest to hardest, then reading them in order, makes very little difference at the end. The textbooks you read in the final two weeks, when you are calm and your memory is freshest, do matter. Pretraining works the same way. Sorting the whole corpus by difficulty is mostly wasted effort, but the data placed in the last few percent of training, during the LR decay phase, is the data the model retains most strongly.

Key concepts

Pretraining

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The curriculum versus annealing debate is one of the cleanest examples of how human-education intuitions break down at LLM scale. Sorting study material from easy to hard works for a student because the student has a small budget of attention and a working memory that benefits from incremental challenge. A 10T-token pretraining run has neither of those constraints. What it does have is a learning rate schedule that makes some steps far more influential than others, and that is the lever that actually pays off.

This walkthrough develops the LR-weighted view of training, fits it to what published recipes do, and closes with the practical decision the PM in the question should have made instead.

Why strict curricula struggle at LLM scale

A strict global curriculum requires three things to work: a defensible difficulty signal, an ordering that improves learning more than shuffling, and a pipeline cost worth the eval gain. All three are weak at scale.

The difficulty signal is the first problem. What does 'difficulty' even mean for a mixed-source corpus of web pages, code, math papers, Wikipedia, books, and Reddit? Document length is not difficulty. Token rarity is not difficulty. Loss-under-a proxy model is closer but expensive to compute at 10T scale and depends on the proxy's own quirks. No simple signal has emerged as robustly useful across published runs.

The ordering benefit is the second problem. Even when a defensible signal exists, comparisons against shuffled-mixture baselines have produced small, inconsistent wins at LLM scale. The model is processing trillions of tokens in batches of millions; the law of large numbers smooths out most ordering effects across the bulk of training.

The pipeline cost is the third problem. Sorting 10T tokens by any signal that depends on content (not just metadata) is a non-trivial engineering project that competes for budget with other improvements. The opportunity cost is real.

The LR-weighted view of training

What labs actually do

The decision the PM should have made

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama 3 explicitly describes a high-quality annealing data mixture for the final phase of pretraining, with measurably better downstream eval results than no-annealing controls
DeepSeek-V3's two-stage schedule reserves a deliberately curated math and code heavy mixture for the WSD decay phase rather than sorting the full corpus by difficulty

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you decide what data goes into the annealing window?

Pick data that is high quality, low duplication, and aligned with the downstream evals you care about. Common choices: deduplicated technical content, curated math problems, code from quality-vetted repos, instruction-style text. The pool should be 5 to 20 percent of the total token budget.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Picking between 'order matters' and 'order does not matter' as a binary, instead of recognizing the asymmetry between the bulk of training (order weakly matters) and the annealing phase (order strongly matters).

60 second bullets to scan on the way to the call.

Why has strict global curriculum learning shown weak results at LLM scale?
What makes the annealing phase mechanically different from the bulk of training?

Primary sources. Browse if you want the original framing.