**Catastrophic forgetting** is when a model gets better at your fine-tuning task but quietly gets worse at everything else it used to do well. The math reasoning, the general knowledge, the coding, all of that can degrade while your target metric looks great.
The key idea is that forgetting is about lost OUT-of-domain skill. Your in-domain numbers can look perfect precisely because the model is over-specialising. So watching your training loss or your in-domain validation set tells you nothing about what was lost elsewhere.
The only honest measurement is a before-and-after comparison on general benchmarks the training never touched. You run the base checkpoint on a suite like MMLU, ARC, and GSM8K, record the scores, then run the fine-tuned checkpoint on the exact same suite. A drop of a few absolute points is the signal.
The standard tool for this is lm-evaluation-harness, which scores both checkpoints identically so the comparison is fair. That is why answer B is correct.
The other options are all internal training signals. Validation loss, in-domain perplexity, and gradient norms describe how training is going, not what general capability survived it.
**Catastrophic forgetting** is the loss of general capability the base model had, caused by over-fitting to a narrow fine-tuning task. By definition it lives outside your training distribution. So no signal computed on your own data can detect it.
That rules out three of the four options immediately. In-domain perplexity measures target-task fit. Validation loss tracks optimisation, and has no universal threshold. Gradient norms describe training dynamics. None of them looks outside the fine-tuning task.
The correct answer is B. You score the base checkpoint and the fine-tuned checkpoint on the same external general-capability suite, then compare. A drop of two to three absolute points on benchmarks like MMLU, ARC, or GSM8K is the standard red flag.
The measurement discipline matters as much as the metric. Use the same harness, the same prompts, and the same few-shot setup for both checkpoints. lm-evaluation-harness is the field standard precisely because it makes the two runs identical. Otherwise prompt drift, not forgetting, explains your delta.
The deeper point: forgetting is a regression on retained skills. It can only be seen by re-running those skills against a fixed external yardstick.
Quick reference
TL;DR
Catastrophic forgetting is lost general capability. Measure it by scoring base and fine-tuned checkpoints on held-out general benchmarks and comparing the drop.
Imagine a brilliant generalist student who you intensively coach to ace one specific exam. The coaching works, but you worry it crowded out everything else they knew. The only honest way to check is to re-test them on the broad subjects they were already good at, history, science, math, and see if those scores dropped. You cannot tell from how well the coaching sessions themselves went. You have to give them the old, general tests again and compare to their scores before coaching. If history fell from ninety to seventy, the coaching cost them. That re-test on the old material is exactly how we catch a fine-tuned model forgetting what it used to know.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Diagnosing whether the fine-tune learned its target task at all That is in-domain fit, measured by target metrics or in-domain perplexity, not by a general-benchmark diff.
Debugging an unstable training run with exploding or vanishing updates Gradient norms and loss curves are the right tools there; general benchmarks say nothing about optimisation health.
Comparing two models trained with different tokenizers using raw loss Loss values are not comparable across tokenizers, so a loss threshold is meaningless for cross-model judgement.
Quick smoke test during a hyperparameter sweep on a laptop Running a full general-capability suite per trial is expensive; reserve it for candidate checkpoints worth shipping.
Real products, models, and research that use this idea.
1EleutherAI's lm-evaluation-harness is the field-standard tool teams use to score base and fine-tuned checkpoints identically on MMLU, ARC, and GSM8K for before-and-after comparison.
2Hugging Face's Open LLM Leaderboard runs a fixed harness suite so a Llama 4 fine-tune can be diffed against its base on the same general benchmarks.
3Meta's Llama 4 and Mistral release notes report general-benchmark retention alongside task gains, exactly the base-versus-tuned diff this question rewards.
4Teams fine-tuning with Hugging Face PEFT and TRL routinely gate releases on an MMLU regression check against the base checkpoint before shipping the adapter.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QOnce you detect a general-benchmark regression, what mitigations reduce catastrophic forgetting?
Sign in or go Pro to see the approach for this follow-up.
QWhy might reporting a single averaged benchmark score hide forgetting that per-task scores reveal?
Sign in or go Pro to see the approach for this follow-up.
QHow does parameter-efficient fine-tuning like LoRA change the forgetting risk versus full fine-tuning?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Watching training-time signals like loss or in-domain perplexity and assuming low numbers mean capability is retained. Forgetting only shows up on external general benchmarks the training never touched.