A team is SFT'ing a 7B model on a narrow legal-summarisation corpus and watching MMLU drop 4 points by step 5k. They want to add a replay buffer of general instruction data to slow the forgetting. Specify a starting recipe: what general source to pull from, what mix ratio to start with, how to tune it, and the failure modes at both extremes of the ratio.
Free · 2 AI evals / day
**Catastrophic forgetting** is what you are watching. The model is improving on legal summarisation but losing skills it learned during pretraining and earlier instruction tuning, which is why MMLU is sliding.
The standard fix is a replay buffer. You mix a small fraction of general instruction-following examples into every training batch, so the model keeps practising what it used to know while learning the new task.
For the general source, pull from a clean public mix like Tulu 2 or 3, UltraChat-200k, or OpenAssistant. If you have access to the model's own original SFT data, that is even better because the distribution matches what shaped the base.
A sensible starting ratio is roughly 10% general and 90% domain. Run a few short pilots at 5%, 10%, and 15% general, then read both MMLU and your domain metric at each setting. Keep the lowest ratio that holds MMLU inside the tolerance you set, often within one or two points.
The two failure modes sit at opposite ends. Too little replay and forgetting continues unchecked. Too much replay and the batches no longer push hard enough on the legal task, so the domain metric stalls.
**Replay** is a small fixed slice of general instruction data interleaved into your domain SFT batches. It pulls the gradient toward the old distribution every step, which slows the drift that produces catastrophic forgetting.
The source matters more than people expect. The cleanest choice is the model's own previous SFT corpus, because the distribution already matches what the base was tuned on. When that is not available, a clean public instruction mix like Tulu 3, UltraChat-200k, or OpenAssistant is a reasonable substitute. Pre-tokenise both sources and interleave at the row level so every batch is mixed; concatenating a general block before the domain block does almost nothing because the long domain tail overwrites it.
Start at roughly 10% general and 90% domain, then treat the ratio as your main knob. Run short pilots at 5%, 10%, and 15% general, log MMLU and your domain metric on a held-out slice every few hundred steps, and pick the lowest ratio that holds MMLU inside a tolerance you set up front, often within one or two points.
The failure modes are symmetric. Under 5% replay is too thin to refresh the general distributions and MMLU keeps falling. Above 20 to 30% the domain signal gets diluted, the target metric plateaus early, and you have quietly converted a domain SFT into a weak continued-pretraining run.
Quick reference
TL;DR
Mix a small slice of general instruction data, around 10%, into your domain SFT batches and tune the ratio against MMLU until forgetting stays inside tolerance.
Imagine teaching someone to write only legal contracts for six months. After a while they get great at contracts but start fumbling everyday emails. To prevent that, every day you sneak in one ordinary email task between the contract drills. Not too many, or the contract skill stalls. Not too few, or the email skill keeps slipping away. You watch two report cards each week, one for contracts and one for general writing, and slide the daily mix up or down until both numbers look healthy. The same idea works for a model. A small refresh of general examples mixed into every batch reminds the network of what it used to know, while most of the practice still pushes it toward the new specialty.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
The base model already lacked the skill you are trying to defend Replay refreshes what was learned, not what was missing. A capability gap needs targeted training data, not a general mix.
Domain and general data conflict on the same surface like refusal style or output format Mixing the two sources injects contradictory targets, so the model learns an inconsistent policy on the disputed surface.
Short LoRA run on a small adapter where the base weights stay frozen Forgetting risk is much lower when only the adapter moves, so the replay overhead rarely justifies the extra data pipeline complexity.
Quick scan · 2–4 min
Everything important, quickly.
6 min: framing forgetting as a two-loss tradeoff + source choice + row-level interleaving + starting ratio + pilot loop with held-out MMLU + failure modes at both extremes + when replay cannot help.
Real-world examples
Real products, models, and research that use this idea.
1DeepSeek V4 and Qwen 3.5 post-training mix general instruction data into domain SFT stages to limit forgetting on broad reasoning benchmarks.
2Meta's Llama 4 Maverick fine-tuning recipes interleave general instruction batches when teams adapt the model to narrow verticals like legal or medical text.
3Hugging Face TRL and Axolotl expose dataset-mixing weights as a first-class config, so teams can blend a domain corpus with Tulu 3 or UltraChat-200k in one YAML.
4Open-source legal and medical adapter projects on Hugging Face commonly publish a 10 to 15% Tulu replay slice alongside their domain split in 2026 model cards.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide the replay ratio if you cannot run multiple pilot fine-tunes?
Sign in or go Pro to see the approach for this follow-up.
QWhy does row-level interleaving beat concatenating a general block before the domain block?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes if you replace SFT with DPO at the next stage, do you still need replay?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Concatenating one block of general data before the domain block. The model sees the general examples first, then forgets them anyway during the long domain tail. Interleave at the row level instead.