Domain eval drift appears mid run: should mix weights change?
At step 520k, coding evals plateau while general QA keeps improving. Two repeated ablations show a +4 coding gain when code tokens are up weighted, with no optimizer instability signals. What action should the training team take?
With clean optimizer telemetry and a code gain that two ablations reproduce, the stall is a data mix problem: up-weight code and guard the other domains against regressions.
Think of a student with a fixed amount of study time each week, split between math and reading. Their reading grades keep rising, but their math grades have stalled. They sleep well and are not burned out, so working harder or resting more is not the fix. Twice, they tried a week with more math time, and both times their math score went up. The sensible move is to shift more of the weekly hours to math. Total hours are fixed, so reading gets a bit less time. They keep an eye on reading grades and shift back if those start to fall.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Mid-run, a team sees coding evals stall while general QA keeps improving. Optimizer telemetry is clean, and two repeated ablations show that giving code more weight lifts coding scores by about 4 points. The right move is to reweight the data mix toward code and keep guardrails on the other domains.
That decision rests on reading the signals in the right order: rule out training health, confirm the effect is real, then trace it to the one variable the ablations changed. The change itself also has costs that need a plan before it goes into a long run.
How the mix shapes each update
A pretraining corpus is a set of domains: web text, books, code, math, multilingual text and others. The data mix gives each domain a sampling weight w_d. Each batch draws tokens by those weights, so the training loss is a weighted average of the loss on each domain:
The gradient follows the same weights. A domain with twice the weight pulls twice as hard on every update, and the model spends more of its capacity on it. Under a fixed token budget D, domain d gets about w_d × D tokens, so raising one weight always lowers others.
Weights are not set in proportion to how much data exists. GPT-3 gave Wikipedia a 3% weight, which meant 3.4 passes over it. Filtered Common Crawl got 60% of the mix and only 0.44 passes. Modern runs pick weights with experiments on small models. Llama 3 used scaling law experiments to choose its mix of about 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual. A mix chosen at the start is a prediction, and mid-run evals can show that it was wrong for one domain.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 chose its final mix (about 50% general knowledge, 25% math and reasoning, 17% code, 8% multilingual) with scaling law experiments on small models, and judged small domain datasets by annealing a half-trained 8B model on 40B tokens with 30% weight on the new data.
- DoReMi tuned domain weights with a 280M proxy model; an 8B model trained on those weights gained 6.5 points of average few-shot accuracy over The Pile's default weights and reached baseline accuracy in 2.6 times fewer steps.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow long and how many seeds should the branch ablations have before changing a 1M-step run?
Weigh branch length against cost. Ask whether a 20k-step branch shows effects that persist, and how many seeds separate a 4-point gain from eval noise.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating a domain plateau as an optimizer problem and changing the learning rate, when stable telemetry and replicated ablations point to the data mix.
60 second bullets to scan on the way to the call.
Training loss as a weighted sum of domain losses
Telemetry that rules out an optimization problem
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.