Predict the eval shifts after tripling the code share of the mixture

Predict output·Medium·4.0 · 0·~2 min·Asked atCerebras Intel Runway

Attempt it

Two pretraining runs share an identical 2T token budget, model size, and hyperparameters. Run A uses 10% code in the data mixture; Run B raises code to 30%, displacing general web text. Predict the direction of change for Run B relative to Run A on: (a) code generation benchmarks like HumanEval, and (b) held out perplexity on general English web text.

Pretraining mixtures are a budget problem, not a capabilities wishlist. At 2T tokens, every percentage point of code share displaces a percentage point of something else, usually general web text. When Run B raises code from 10% to 30%, two things happen. On **code benchmarks** like HumanEval and MBPP, scores improve. The model sees three times more code tokens, which is roughly a three-fold increase in in-distribution training for that domain. Improvements of several points on HumanEval are typical for this kind of shift at this scale. On **held-out English web perplexity**, the score gets worse. Web perplexity is measured against general web text, and Run B trained on substantially fewer of those tokens. The degradation is usually modest (a few percent relative), not catastrophic, because 30 percent code still leaves 70 percent for other sources. The interesting nuance is that code exposure is often credited with helping some reasoning evals (GSM8K, BBH) too, so the picture is not purely a code vs web trade. But for the two specific evals in the question, the prediction is clear: code up, web perplexity worse.

Mixture ratios at fixed compute are zero-sum. Tripling code displaces other tokens, and the eval shifts follow that displacement. - **Code benchmarks (HumanEval, MBPP).** Improve, typically substantially. Run B sees 600B code tokens to Run A's 200B, which is a 3x increase in in-distribution training for this domain. Empirical reports from Llama 3 and DeepSeek mixture ablations show that code benchmark gains are roughly monotone with code share until quality saturates. - **Held-out English web perplexity.** Degrades, typically modestly. Run B trains on roughly 22 percent fewer web tokens (if code displaced web 1-for-1), so perplexity on that held-out distribution rises. The effect is usually a few percent relative, not a step change, because absolute token count for web is still very large. Two nuances worth naming. First, code exposure has been credited in several papers with helping reasoning benchmarks even when measured outside the code domain, so the trade is not purely 'code up, everything else down'. Second, what matters in production is usually downstream evals (MMLU, HumanEval, your own benchmark), not raw held-out perplexity; teams accept small perplexity hits when the downstream picture improves. The right operational posture is to run a small-scale mixture ablation rather than guess. The Chinchilla-era practice of fixing one ratio and shipping has been replaced by per-budget mixture sweeps.

TL;DR

At a fixed token budget, mixture ratios are zero-sum: tripling code share lifts code evals and degrades held-out web perplexity in proportion to the displacement.

Memory aid

Easy to grasp

Imagine packing a single suitcase for a two-week trip. You decide to bring three times as many running shoes. The suitcase did not get bigger, so something else (work shirts, a jacket) had to come out. You will run great. You will look worse at meetings. Pretraining mixtures work the same way: the token budget is the suitcase. Tripling the code share means less room for general web text, so the model gets better at writing code and a little worse at modeling everyday English. Neither result is surprising once you remember the suitcase is full.

Key concepts

Pretraining

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Mixture-ratio prediction questions look like they are about a specific domain, but they are really about understanding the fixed-budget constraint that makes pretraining a planning problem rather than a sampling problem. Once you internalize that constraint, the answer to most mixture questions falls out without needing benchmark numbers.

This walkthrough develops the fixed-budget intuition, fits it to the actual functional form of pretraining loss, and then layers on the second-order effects (quality saturation, transfer to non-domain evals) that distinguish a careful answer from a casual one.

The fixed-budget argument

At a 2T token budget, the mixture ratios literally allocate the budget. Run A spends 200B tokens on code and 1.8T on everything else. Run B spends 600B on code and 1.4T on everything else. If code displaces web one for one, Run B's web allocation is 22 percent smaller than Run A's.

This is the part most candidates skip. They predict code-up confidently because more code is in, but they forget that the suitcase did not get bigger. The web tokens that used to be in the suitcase are now gone. Some evaluation surface is going to feel that, and held-out web perplexity is exactly the surface that does.

The move that makes the question easy is to write down both sides of the ledger explicitly: code tokens up 3x, web tokens down 22 percent, and then ask what each shift implies for the corresponding eval.

The functional form: power laws, not proportional moves

Second-order effects that flip the easy answer

Why labs ablate instead of guessing

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

# Reweight the mixture: triple the code share, displace web proportionally.
mix = {"web": 0.75, "code": 0.10, "math": 0.08, "books": 0.07}
mix["code"] *= 3                       # 0.10 -> 0.30
shortfall = 1.0 - sum(mix.values())    # negative: we overcommitted
mix["web"] += shortfall                # absorb displacement on web
assert abs(sum(mix.values()) - 1.0) < 1e-9
print({k: round(v, 4) for k, v in mix.items()})
# {'web': 0.55, 'code': 0.30, 'math': 0.08, 'books': 0.07}

Real products, models, and research that use this idea.

Llama 3's pretraining paper reports explicit mixture ablations across code, web, math, and multilingual buckets, with each ablation tracking the in-distribution gain against the displaced-domain loss
DeepSeek-V3's FP8 training mixture mid-shifted toward code and math during the latter phase of training, accepting small web-perplexity costs for substantial HumanEval and GSM8K gains

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design a mixture ablation to find the optimal code share for an 8B model?

Train 4 to 6 small proxy runs at fixed compute with code share at 5, 10, 20, 30, 50 percent, measure HumanEval and held-out perplexity per source, fit the tradeoff curve, and pick the share where marginal HumanEval gain equals marginal cost on the downstream evals you care about.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Predicting that more code helps everything because code 'teaches reasoning', without accounting for the web tokens that had to be displaced at a fixed budget.

60 second bullets to scan on the way to the call.

Why is a pretraining mixture a zero-sum allocation at fixed token budget?
What is the typical functional form of perplexity versus tokens seen on a domain?

Primary sources. Browse if you want the original framing.