Which of these reduce catastrophic forgetting in practice?

Attempt it

Quick reference

TL;DR

Forgetting is gradient drift away from pretrained weights. Fight it by moving less: low LR, few epochs, frozen-base PEFT, and a replay buffer of general data.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a skilled chef who already cooks hundreds of dishes. You want to teach them one new regional cuisine. If you drill that one cuisine intensely day and night for weeks, they get great at it but start fumbling their old recipes. That is catastrophic forgetting. The safe approach is gentler. Give short lessons instead of marathons. Make small tweaks rather than retraining the chef from scratch. Add a thin new recipe card on top of their existing skills instead of rewriting their whole brain. And keep cooking a few old favourites during the lessons so those skills stay warm. The point is always the same: nudge the chef a little toward the new thing while protecting everything they already knew.

Key concepts

Catastrophic forgetting LoRA / QLoRA Replay / rehearsal data Learning rate and epochs

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: forgetting as weight drift + low LR and fewer epochs + frozen-base PEFT + replay buffer + why batch size and epochs are traps + measuring on held-out benchmarks.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT documents LoRA and QLoRA as the default way to fine-tune Llama 4 and Qwen without overwriting pretrained capability.
Continual-learning research on Elastic Weight Consolidation pioneered the rehearsal and parameter-importance penalties now reused in LLM fine-tuning.
Axolotl and Unsloth recipes ship one-to-three epoch defaults with low peak learning rates specifically to limit forgetting on small datasets.
Teams routinely re-run MMLU, ARC, and HellaSwag before and after fine-tuning a DeepSeek V4 or Mistral base to detect capability regressions.
OpenAI's fine-tuning guide recommends small epoch counts and conservative learning-rate multipliers on GPT-4o-mini to preserve base behavior.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does LoRA preserve general capability even when the adapter is trained aggressively?

Sign in or go Pro to see the approach for this follow-up.

QHow does Elastic Weight Consolidation reduce forgetting, and why is it rarely used for LLM fine-tuning?

Sign in or go Pro to see the approach for this follow-up.

QHow would you quantify forgetting rather than assume it?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking bigger batches or more epochs help. They do not. Both push the model harder on the narrow domain, which accelerates forgetting rather than slowing it.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.