How do you detect catastrophic forgetting after a domain fine-tune?

**Catastrophic forgetting** is when fine-tuning a model on your narrow data makes it worse at everything else it used to know. You taught it legal drafting, and now it fumbles basic arithmetic. The only reliable way to catch this is to measure, not eyeball. Pick a suite of broad benchmarks that cover different skills: MMLU for general knowledge, ARC and HellaSwag for reasoning, GSM8K for math, and HumanEval for code. These are your general-capability checks. Run that exact suite twice: once on the original base model, once on your fine-tuned model. Then compare the scores side by side. The number you care about is the delta, how much each score moved. A drop of more than 2 to 3 absolute points on the average is a real warning sign. A drop above 5 points is severe and usually means your learning rate was too high or you trained for too many epochs. The standard tool for this is EleutherAI's lm-evaluation-harness, because it scores every benchmark the same way the rest of the community does, so your numbers are comparable to published baselines. Also run your own in-domain test, the forgetting check only proves the fine-tune was safe, not that it worked.

**Catastrophic forgetting** is measured, never eyeballed. You build a general-capability eval suite and run it on two checkpoints. The suite should span multiple capability axes. Use MMLU for knowledge, ARC-Challenge and HellaSwag for common-sense reasoning, GSM8K for math, and HumanEval for code if the base supports it. Hold this suite completely out of the training data. Run the identical suite on the base checkpoint and the fine-tuned checkpoint. The signal is the delta on the averaged score, not the absolute number. A drop above 2 to 3 absolute points is a meaningful forgetting signal. A drop above 5 points is severe and points to over-aggressive hyperparameters or overly narrow domain data. The canonical tool is EleutherAI's lm-evaluation-harness. It applies consistent prompting and scoring, so your delta is directly comparable to published numbers. Run the base and fine-tuned models through the same config to keep the comparison honest. Two caveats matter. First, drops under 2 points are often within run-to-run noise, so do not over-react. Second, the forgetting check only proves safety. You still need an in-domain eval to prove the fine-tune actually bought you the capability you wanted.