Pick the mid run eval design that gives real signal at 1T tokens
Use smooth proxies like per-domain held-out loss and easy few-shot tasks with continuous metrics, compared against a trusted reference run at matched token counts.
Imagine baking a giant cake that takes 12 hours, and at hour one you want to know if it is going well. You cannot taste the final flavor yet, that comes near the end. So you check things that change smoothly the whole way: how high it has risen, what color it is turning, what temperature the center reads. You also have a recipe card from a cake you trusted last week, with notes about what each thing should look like at hour one. If your cake's measurements match the trusted card, the bake is on track. Hard benchmarks are like asking whether the cake tastes good at hour one. The honest answer is that it tastes like raw batter, which tells you nothing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A pretraining run at 1T of a planned 12T tokens is the worst possible moment to be uninformed about run health, and also the moment when the most attractive-looking metrics are the least informative. The wrong eval design at 1T can kill a healthy run or rubber-stamp a sick one. The right design exploits the difference between metrics that move smoothly across scale and metrics that emerge only late.
This deep dive covers what makes a metric smooth, why hard benchmarks read as noise at 1T, the three-layer mid-run dashboard, the value of a reference-run comparison, and when it is worth spending compute on the full benchmark suite. The takeaway: routine mid-run eval is about smooth proxies against a baseline; expensive evals are reserved for designated decision points.
Continuous versus emergent metrics
A metric is continuous if its value moves with training compute from the first step. Per-token held-out loss is the cleanest example: every gradient step that helps the model shows up in the loss curve. Log-likelihood multiple choice on easy tasks is another. LAMBADA accuracy, HellaSwag log-likelihood, ARC-easy: all smooth, all readable early.
A metric is emergent if its value sits at chance for most of training and transitions sharply later. MATH, GPQA, MMLU at the harder slices, code generation accuracy on serious tasks. These metrics have a phase transition behaviour: random until a capability threshold is crossed, then climbing rapidly. The threshold often lies well past 1T tokens for a 70B-class model on a 12T-token budget.
This is not a flaw in the metrics; it reflects the real structure of what they measure. But it means reading them mid-run treats noise as signal. A 25 percent MATH score at 1T tokens is what you get from random four-choice guessing. Calling the run a failure on that basis is a category error.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 and OLMo 2 training reports document per-domain validation tracking against reference scales throughout pretraining.
- Anthropic's Claude release notes describe pre-anneal decision checkpoints distinct from routine mid-run health monitoring.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour per-domain loss on code is flat while web and books continue to drop. What do you do?
Treat as a real regression. Check the mixture weights, recently added data sources, and tokenizer behaviour on code. Compare against the reference run's code curve at the same token count. If the gap is widening, intervene before the gap consolidates.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading hard-benchmark scores at 1T tokens as a quality signal when those scores sit at chance and only emerge much later in the run.
60 second bullets to scan on the way to the call.
What is an emergent benchmark and why does it read as noise mid-run?
How does per-domain held-out loss surface problems global loss misses?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.