What are the eval shifts after tripling the code share of the mixture

Predict output·Medium·4.0 · 0·~2 min·Asked atCerebras Intel Runway

Attempt it

Two pretraining runs share an identical 2T token budget, model size, and hyperparameters. Run A uses 10% code in the data mixture; Run B raises code to 30%, displacing general web text. What is the direction of change for Run B relative to Run A on: (a) code generation benchmarks like HumanEval, and (b) held out perplexity on general English web text.

Pretraining mixtures are a budget problem, not a capabilities wishlist. At 2T tokens, every percentage point of code share displaces a percentage point of something else, usually general web text. When Run B raises code from 10% to 30%, two things happen. On **code benchmarks** like HumanEval and MBPP, scores improve. The model sees three times more code tokens, which is roughly a three-fold increase in in-distribution training for that domain. Improvements of several points on HumanEval are typical for this kind of shift at this scale. On **held-out English web perplexity** (a measure of prediction error), the score gets worse. Web perplexity is measured against general web text, and Run B trained on substantially fewer of those tokens. The degradation is usually modest (a few percent relative), not catastrophic, because 30 percent code still leaves 70 percent for other sources. Code exposure is often credited with helping some reasoning evals (GSM8K, BBH) too, so the picture is not purely a code vs web trade. But for these two evals, the prediction is clear: code up, web perplexity worse.

Mixture ratios at fixed compute are zero-sum. Tripling code displaces other tokens, and the eval shifts follow that displacement. - **Code benchmarks (HumanEval, MBPP).** Improve, typically substantially. Run B sees 600B code tokens to Run A's 200B, which is a 3x increase in in-distribution training for this domain. Mixture ablations generally show code benchmark gains rising with code share until quality saturates. - **Held-out English web perplexity.** Degrades, typically modestly. Non-code tokens fall from 1.8T to 1.4T. If the cut is spread across all non-code sources, Run B sees about 22 percent fewer web tokens, and more if web alone absorbs it. Perplexity on that held-out distribution rises. The effect is usually a few percent relative, not a step change, because absolute token count for web is still very large. Two caveats apply. First, several papers credit code exposure with helping reasoning benchmarks even when measured outside the code domain. So the trade is not purely 'code up, everything else down'. Second, what matters in production is usually downstream evals (MMLU, HumanEval, your own benchmark), not raw held-out perplexity. Teams accept small perplexity hits when the downstream picture improves. The right approach is to run a small-scale mixture ablation rather than guess. Llama 3, for example, chose its final mixture with scaling law experiments on small models trained on candidate mixes.

TL;DR

At a fixed token budget, mixture ratios are zero-sum: tripling code share lifts code evals and modestly degrades held-out web perplexity, which follows a power law in tokens seen.

Memory aid

Easy to grasp

Imagine packing a single suitcase for a two-week trip. You decide to bring three times as many running shoes. The suitcase did not get bigger, so something else (work shirts, a jacket) had to come out. You will run great. You will look worse at meetings. Pretraining mixtures work the same way: the token budget is the suitcase. Tripling the code share means less room for general web text, so the model gets better at writing code and a little worse at modeling everyday English. Neither result is surprising once you remember the suitcase is full.

Key concepts

Pretraining

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Mixture ratio predictions look like they are about a specific domain. But they rest on the fixed-budget constraint that makes pretraining a planning problem rather than a sampling problem. Once you understand that constraint, most mixture predictions follow without needing benchmark numbers.

The fixed-budget argument

At a 2T token budget, the mixture ratios allocate the budget. Run A spends 200B tokens on code and 1.8T on everything else. Run B spends 600B on code and 1.4T on everything else. If the cut is spread across all non-code sources, Run B's web allocation is 22 percent smaller than Run A's. If web alone absorbs it, the web cut is larger.

A common error is to predict code up confidently because more code is in, and to forget that the token budget did not get bigger. The web tokens that the extra code replaced are now gone. Some evaluation surface is going to feel that, and held-out web perplexity is exactly the surface that does.

The fix is to write down both sides of the ledger explicitly: code tokens up 3x and web tokens down 22 percent. Then ask what each shift implies for the corresponding eval.

The functional form: power laws, not proportional moves

Second-order effects that flip the easy answer

Why labs ablate instead of guessing

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

# Reweight the mixture: triple the code share and take the extra from web.
mix = {"web": 0.75, "code": 0.10, "math": 0.08, "books": 0.07}
mix["code"] *= 3                       # 0.10 -> 0.30
shortfall = 1.0 - sum(mix.values())    # negative: the mix is overcommitted
mix["web"] += shortfall                # absorb displacement on web
assert abs(sum(mix.values()) - 1.0) < 1e-9
print({k: round(v, 4) for k, v in mix.items()})
# {'web': 0.55, 'code': 0.30, 'math': 0.08, 'books': 0.07}

Real products, models, and research that use this idea.

Llama 3 chose its final mixture (roughly 50 percent general knowledge, 25 percent math and reasoning, 17 percent code, 8 percent multilingual) using scaling law experiments on small models trained on candidate mixes
DeepSeek-V3's report says it raised the ratio of math and programming samples in its pretraining corpus compared with DeepSeek-V2, while expanding multilingual coverage

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design a mixture ablation to find the optimal code share for an 8B model?

Train 4 to 6 small proxy runs at fixed compute with code share at 5, 10, 20, 30, 50 percent. Measure HumanEval and held-out perplexity per source, and fit the tradeoff curve. Pick the share where marginal HumanEval gain equals marginal cost on the downstream evals you care about.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Predicting that more code helps everything because code 'teaches reasoning', without accounting for the web tokens that had to be displaced at a fixed budget.

60 second bullets to scan on the way to the call.

Why is a pretraining mixture a zero-sum allocation at fixed token budget?
What is the typical functional form of perplexity versus tokens seen on a domain?

Primary sources. Browse if you want the original framing.