How much general data should a replay buffer for domain SFT use, and from which general dataset?
A team is SFT'ing a 7B model on a narrow legal summarisation corpus and watching MMLU drop 4 points by step 5k. They want to add a replay buffer of general instruction data to slow the forgetting. Specify a starting mix: what general source to pull from, what mix ratio to start with, how to tune it, and the failure modes at both extremes of the ratio.
Mix a small slice of general instruction data, around 10%, into your domain SFT batches and tune the ratio against MMLU until forgetting stays inside tolerance.
Imagine teaching someone to write only legal contracts for six months. After a while they get great at contracts but start fumbling everyday emails. To prevent that, every day you sneak in one ordinary email task between the contract drills. Do not add too many, or the contract skill stalls. Do not add too few, or the email skill keeps slipping away. You watch two report cards each week, one for contracts and one for general writing. You slide the daily mix up or down until both numbers look healthy. The same idea works for a model. A small refresh of general examples mixed into every batch reminds the network of what it used to know. Most of the practice still pushes it toward the new specialty.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Catastrophic forgetting during domain SFT is not a sign that something has gone wrong. It is the predictable consequence of pushing a model hard on a narrow distribution for thousands of steps. Every gradient step moves the weights toward the local task and away from whatever the base was good at. On a long enough run, the model learns the new task and quietly loses skills it used to have. The 4-point MMLU drop by step 5k is the symptom. The cause is that the batches contain almost nothing the model needs to remember.
A replay buffer is the standard, cheap fix. You set aside a small fraction of general instruction-following data and interleave it into every training batch. The domain examples still dominate the gradient direction, but the general examples pull the weights back toward the broader distribution on every step. The trick is choosing the right source, the right ratio, the right interleaving, and the right metric to tune against. Get any of those wrong and you either keep forgetting (too little replay, wrong source) or stall the domain task (too much replay).
Why narrow SFT causes broad forgetting
Pretrained models hold a wide distribution of skills, encoded across billions of parameters. Each fine-tuning step is a gradient that says move toward this kind of output. When every example for thousands of steps belongs to the same narrow domain, the gradient consistently pushes the weights toward one corner of parameter space. It also pushes them away from the configuration that supported broader competence.
The score on the held-out broad benchmark does not drop because the model has been told to be worse at it. It drops because nothing in the data is pushing back. The general capabilities have no protection in the gradient. They sit downstream of weights that are being repurposed for the new task, and they degrade as a side effect.
This is why training loss alone does not show the problem. Training loss tells you the model is fitting the domain data, which is exactly what it should be doing. The damage shows up only when you evaluate on something the training data does not cover. That is also why a sliding MMLU is a high-trust signal. It measures skills the model is not currently being asked to display. So a regression there is an unintended side effect of the optimisation, not something it was aiming for.
Replay is the simplest counterweight. By injecting a small slice of general data into every batch, you give the broad skills a small but steady gradient term. That term is enough to slow the drift without taking over the optimisation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Code Llama kept 7% general natural language in its code training to retain natural language understanding. This is the same replay idea at continued-pretraining scale.
- Rehearsal methods in continual learning keep stored examples of earlier tasks in training. They are the origin of the replay buffer idea used in domain SFT.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide the replay ratio if you cannot run multiple pilot fine-tunes?
Estimate from a short warmup. Run a single pilot at 10% for a few thousand steps and watch the slope of the held-out MMLU curve. If it is flat or rising, hold the ratio. If it is still falling, double the replay fraction once and re-evaluate.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Concatenating one block of general data before the domain block. The model sees the general examples first, then forgets them anyway during the long domain tail. Interleave at the row level instead.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.