Which signal is the most reliable indicator of catastrophic forgetting?

**Catastrophic forgetting** is when a model gets better at your fine-tuning task but quietly gets worse at everything else it used to do well. The math reasoning, the general knowledge, the coding, all of that can degrade while your target metric looks great. The key idea is that forgetting is about lost OUT-of-domain skill. Your in-domain numbers can look perfect precisely because the model is over-specialising. So watching your training loss or your in-domain validation set tells you nothing about what was lost elsewhere. The only honest measurement is a before-and-after comparison on general benchmarks the training never touched. You run the base checkpoint on a suite like MMLU, ARC, and GSM8K, record the scores, then run the fine-tuned checkpoint on the exact same suite. A drop of a few absolute points is the signal. The standard tool for this is lm-evaluation-harness, which scores both checkpoints identically so the comparison is fair. That is why answer B is correct. The other options are all internal training signals. Validation loss, in-domain perplexity, and gradient norms describe how training is going, not what general capability survived it.

**Catastrophic forgetting** is the loss of general capability the base model had, caused by over-fitting to a narrow fine-tuning task. By definition it lives outside your training distribution. So no signal computed on your own data can detect it. That rules out three of the four options immediately. In-domain perplexity measures target-task fit. Validation loss tracks optimisation, and has no universal threshold. Gradient norms describe training dynamics. None of them looks outside the fine-tuning task. The correct answer is B. You score the base checkpoint and the fine-tuned checkpoint on the same external general-capability suite, then compare. A drop of two to three absolute points on benchmarks like MMLU, ARC, or GSM8K is the standard red flag. The measurement discipline matters as much as the metric. Use the same harness, the same prompts, and the same few-shot setup for both checkpoints. lm-evaluation-harness is the field standard precisely because it makes the two runs identical. Otherwise prompt drift, not forgetting, explains your delta. The deeper point: forgetting is a regression on retained skills. It can only be seen by re-running those skills against a fixed external yardstick.