Picking a cheap but predictive eval to track during training: what do you actually log?

**Mid-training evaluation** is a triage problem. You cannot afford to run a slow judge model on every checkpoint, you cannot trust raw training loss because it just measures fit to the training data, and full benchmarks take hours. The compromise is a cheap held-out metric that correlates well with the end of run quality you actually care about. The recipe is concrete. Set aside 200 to 500 examples from your SFT data before training starts. These examples come from the same task distribution but are never used for gradient steps. Every few hundred steps, run a single forward pass on this held-out slice and log the token-level cross-entropy on the assistant tokens only, the same masking pattern as training loss. This is cheap because it is just one forward pass per example, no generation, no judge model. It is deterministic because there is no sampling, so curves from two runs can be overlaid and compared directly. It correlates with downstream quality because lower cross-entropy on held-out responses means the model is assigning higher probability to the kinds of replies your data marks as correct, which is what shows up as better completions at decode time. What the metric does not catch is cross-task regression. Your held-out task cross-entropy can fall while MMLU silently slides 4 points. Pair with an occasional small broad eval, and reserve heavy judge runs for end of run promotion.

**The recipe is a 200 to 500 example held-out slice of the SFT data**, drawn from the same task distribution and never seen during training. Every two to five hundred steps, run a single forward pass on this slice and log token-level cross-entropy on assistant tokens only, masking user and system turns exactly the way training loss masks them. This metric is cheap by construction: no generation, no judge model, no decoding loop, just one forward pass per example. A hundred examples at five hundred tokens each takes seconds on training hardware, so it slots cleanly between training steps without slowing throughput. It is also deterministic, no sampling, no temperature, so you can overlay curves from two different runs and compare them as is. The correlation with downstream quality is the key property. Within the same task family, lower held-out cross-entropy means the model is placing more probability mass on the response tokens your data labels as correct. That mass shows up at decode time as better-ranked completions and higher judge scores. The relationship is not perfect but it is strong enough to call regressions early: a held-out CE that is flat or rising for a thousand steps is a serious signal even if training loss looks fine. What this proxy does not catch is cross-task generalisation. A SFT run can show a beautifully falling held-out CE on the target task while quietly losing four points of MMLU on broad capabilities. Pair the proxy with an occasional small broad eval, say a 200-example MMLU slice every few thousand steps, and reserve the full judge-based suite for end of run promotion decisions.