You've just finished a domain fine-tune of a 7B model. How do you check whether you've induced catastrophic forgetting on general capability? What benchmarks would you run, what's the typical 'concerning' delta threshold, and what tool would you use?
Free · 2 AI evals / day
**Catastrophic forgetting** is when fine-tuning a model on your narrow data makes it worse at everything else it used to know. You taught it legal drafting, and now it fumbles basic arithmetic.
The only reliable way to catch this is to measure, not eyeball. Pick a suite of broad benchmarks that cover different skills: MMLU for general knowledge, ARC and HellaSwag for reasoning, GSM8K for math, and HumanEval for code. These are your general-capability checks.
Run that exact suite twice: once on the original base model, once on your fine-tuned model. Then compare the scores side by side. The number you care about is the delta, how much each score moved.
A drop of more than 2 to 3 absolute points on the average is a real warning sign. A drop above 5 points is severe and usually means your learning rate was too high or you trained for too many epochs.
The standard tool for this is EleutherAI's lm-evaluation-harness, because it scores every benchmark the same way the rest of the community does, so your numbers are comparable to published baselines. Also run your own in-domain test, the forgetting check only proves the fine-tune was safe, not that it worked.
**Catastrophic forgetting** is measured, never eyeballed. You build a general-capability eval suite and run it on two checkpoints.
The suite should span multiple capability axes. Use MMLU for knowledge, ARC-Challenge and HellaSwag for common-sense reasoning, GSM8K for math, and HumanEval for code if the base supports it. Hold this suite completely out of the training data.
Run the identical suite on the base checkpoint and the fine-tuned checkpoint. The signal is the delta on the averaged score, not the absolute number. A drop above 2 to 3 absolute points is a meaningful forgetting signal. A drop above 5 points is severe and points to over-aggressive hyperparameters or overly narrow domain data.
The canonical tool is EleutherAI's lm-evaluation-harness. It applies consistent prompting and scoring, so your delta is directly comparable to published numbers. Run the base and fine-tuned models through the same config to keep the comparison honest.
Two caveats matter. First, drops under 2 points are often within run-to-run noise, so do not over-react. Second, the forgetting check only proves safety. You still need an in-domain eval to prove the fine-tune actually bought you the capability you wanted.
Quick reference
TL;DR
Run a general-capability suite (MMLU, ARC, GSM8K, HumanEval) on the base AND the fine-tuned checkpoint, then compare. A drop above 2-3 absolute points means forgetting.
Imagine a brilliant generalist student who you send to a six-week course on tax law. They come back great at tax, but did they forget basic history, math, and how to write a normal email? The only honest way to find out is to give them the same broad pop quiz BEFORE the course and AFTER it, then compare the two scorecards subject by subject. If the tax score jumped but the math and history scores barely moved, the course was safe. If history crashed while tax soared, the course overwrote things you wanted to keep. Fine-tuning a model is identical: you score the original model on a wide quiz, fine-tune it, score it again on the exact same quiz, and watch the gap.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A tiny LoRA adapter at low rank trained for a fraction of an epoch on a few hundred examples Movement from the frozen base is so small that a full benchmark battery is overkill; a quick MMLU spot-check suffices.
The base model never had the capability you are benchmarking, such as code on a text-only base A near-zero score on HumanEval before and after is not forgetting; do not include axes the base cannot do.
You only care about a single narrow deployment surface with no out of domain traffic If users never send general prompts, in-domain eval matters more than a broad capability regression sweep.
Rapid iteration loops where a full multi-benchmark run costs hours per checkpoint Run a cheap MMLU subset during iteration and reserve the full suite for release-candidate checkpoints.
Quick scan · 2–4 min
Everything important, quickly.
4 min: multi-axis suite + base-vs-fine-tuned delta + concerning and severe thresholds + noise floor + lm-evaluation-harness + LoRA and replay mitigations + in-domain eval caveat.
Side-by-side
Average delta vs base
Interpretation
Likely cause / action
Within +/- 2 points
Noise; no forgetting signal
Ship; differences are run to run variance
2 to 3 point drop
Meaningful forgetting
Lower LR or epochs; add replay data
Above 5 point drop
Severe forgetting
Hyperparameters too aggressive or corpus too narrow
In-domain up, general flat
Healthy fine-tune
Desired outcome; behavior gained, capability kept
Real-world examples
Real products, models, and research that use this idea.
1EleutherAI's lm-evaluation-harness is the de facto standard for running MMLU, ARC, GSM8K and HumanEval with consistent prompting across base and fine-tuned checkpoints.
2Hugging Face's Open LLM Leaderboard scores submitted fine-tunes on the same harness benchmarks, so teams compare their delta against published base-model numbers like Llama 4 and Qwen 3.
3Unsloth and Axolotl fine-tuning configs commonly recommend mixing a slice of general instruction data (FLAN, OpenOrca) back into the training set as replay to limit forgetting.
4Anthropic and OpenAI publish capability-regression suites alongside Claude Opus 4.7 and GPT-5.5 fine-tuning APIs so customers can verify a domain fine-tune did not erode general performance.
5DeepSeek V4 and Llama 4 release notes report MMLU and GSM8K deltas across instruction-tuning stages to demonstrate retained general capability.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is a 1.5-point MMLU drop usually ignored while a 3-point drop is flagged?
Sign in or go Pro to see the approach for this follow-up.
QHow does LoRA mechanically reduce forgetting, and where does it still leak through?
Sign in or go Pro to see the approach for this follow-up.
QHow would you design a replay mixing ratio to counter forgetting without diluting the domain signal?
Sign in or go Pro to see the approach for this follow-up.
QBeyond benchmarks, what regression would a multiple-choice suite miss?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Eyeballing a few sample prompts instead of running a fixed benchmark suite on both checkpoints. Without the base-model baseline you have no delta, so you cannot tell forgetting from noise.