Two pretraining runs share an identical 2T token budget, model size, and hyperparameters. Run A uses 10% code in the data mixture; Run B raises code to 30%, displacing general web text. Predict the direction of change for Run B relative to Run A on: (a) code generation benchmarks like HumanEval, and (b) held-out perplexity on general English web text.
Pretraining mixtures are a budget problem, not a capabilities wishlist. At 2T tokens, every percentage point of code share displaces a percentage point of something else, usually general web text.
When Run B raises code from 10% to 30%, two things happen.
On **code benchmarks** like HumanEval and MBPP, scores improve. The model sees three times more code tokens, which is roughly a three-fold increase in in-distribution training for that domain. Improvements of several points on HumanEval are typical for this kind of shift at this scale.
On **held-out English web perplexity**, the score gets worse. Web perplexity is measured against general web text, and Run B trained on substantially fewer of those tokens. The degradation is usually modest (a few percent relative), not catastrophic, because 30 percent code still leaves 70 percent for other sources.
The interesting nuance is that code exposure is often credited with helping some reasoning evals (GSM8K, BBH) too, so the picture is not purely a code vs web trade. But for the two specific evals in the question, the prediction is clear: code up, web perplexity worse.
Mixture ratios at fixed compute are zero-sum. Tripling code displaces other tokens, and the eval shifts follow that displacement.
- **Code benchmarks (HumanEval, MBPP).** Improve, typically substantially. Run B sees 600B code tokens to Run A's 200B, which is a 3x increase in in-distribution training for this domain. Empirical reports from Llama 3 and DeepSeek mixture ablations show that code benchmark gains are roughly monotone with code share until quality saturates.
- **Held-out English web perplexity.** Degrades, typically modestly. Run B trains on roughly 22 percent fewer web tokens (if code displaced web 1-for-1), so perplexity on that held-out distribution rises. The effect is usually a few percent relative, not a step change, because absolute token count for web is still very large.
Two nuances worth naming. First, code exposure has been credited in several papers with helping reasoning benchmarks even when measured outside the code domain, so the trade is not purely 'code up, everything else down'. Second, what matters in production is usually downstream evals (MMLU, HumanEval, your own benchmark), not raw held-out perplexity; teams accept small perplexity hits when the downstream picture improves.
The right operational posture is to run a small-scale mixture ablation rather than guess. The Chinchilla-era practice of fixing one ratio and shipping has been replaced by per-budget mixture sweeps.
Quick reference
TL;DR
At a fixed token budget, mixture ratios are zero-sum: tripling code share lifts code evals and degrades held-out web perplexity in proportion to the displacement.
Imagine packing a single suitcase for a two-week trip. You decide to bring three times as many running shoes. The suitcase did not get bigger, so something else (work shirts, a jacket) had to come out. You will run great. You will look worse at meetings. Pretraining mixtures work the same way: the token budget is the suitcase. Tripling the code share means less room for general web text, so the model gets better at writing code and a little worse at modeling everyday English. Neither result is surprising once you remember the suitcase is full.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You have headroom in the token budget and are adding code alongside web rather than displacing it The fixed-budget displacement argument does not apply; you are spending more compute, not reallocating it
Your downstream product cares only about code generation and not about general perplexity The web-perplexity degradation is irrelevant to your success metric
You are running a continual pretraining phase rather than a fresh run The replay and mixture dynamics differ; catastrophic forgetting becomes the dominant concern, not the displacement bookkeeping
Quick scan · 2–4 min
Everything important, quickly.
6 minutes: the fixed-budget displacement argument, why perplexity moves as a power law in tokens-seen, the second-order quality-saturation effect, and the operational case for running mixture ablations rather than guessing.
Code example
python
# Reweight the mixture: triple the code share, displace web proportionally.
mix = {"web": 0.75, "code": 0.10, "math": 0.08, "books": 0.07}
mix["code"] *= 3 # 0.10 -> 0.30
shortfall = 1.0 - sum(mix.values()) # negative: we overcommitted
mix["web"] += shortfall # absorb displacement on web
assert abs(sum(mix.values()) - 1.0) < 1e-9
print({k: round(v, 4) for k, v in mix.items()})
# {'web': 0.55, 'code': 0.30, 'math': 0.08, 'books': 0.07}
Real-world examples
Real products, models, and research that use this idea.
1Llama 3's pretraining paper reports explicit mixture ablations across code, web, math, and multilingual buckets, with each ablation tracking the in-distribution gain against the displaced-domain loss
2DeepSeek-V3's FP8 training mixture mid-shifted toward code and math during the latter phase of training, accepting small web-perplexity costs for substantial HumanEval and GSM8K gains
3OLMo 2 published per-source mixture weights and the corresponding held-out perplexity per source, making the allocation tradeoff visible to external researchers
4Qwen 2.5 documentation describes a multi-stage mixture schedule that raises code share later in training to maximize the displacement vs gain ratio
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design a mixture ablation to find the optimal code share for an 8B model?
Sign in or go Pro to see the approach for this follow-up.
QWhy might a 50 percent code share underperform a 30 percent share even on HumanEval?
Sign in or go Pro to see the approach for this follow-up.
QWhen does code exposure help non-code evals like GSM8K, and when is it a red herring?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Predicting that more code helps everything because code 'teaches reasoning', without accounting for the web tokens that had to be displaced at a fixed budget.