Replay-buffer mix for domain SFT: how much general data, and from where?

**Catastrophic forgetting** is what you are watching. The model is improving on legal summarisation but losing skills it learned during pretraining and earlier instruction tuning, which is why MMLU is sliding. The standard fix is a replay buffer. You mix a small fraction of general instruction-following examples into every training batch, so the model keeps practising what it used to know while learning the new task. For the general source, pull from a clean public mix like Tulu 2 or 3, UltraChat-200k, or OpenAssistant. If you have access to the model's own original SFT data, that is even better because the distribution matches what shaped the base. A sensible starting ratio is roughly 10% general and 90% domain. Run a few short pilots at 5%, 10%, and 15% general, then read both MMLU and your domain metric at each setting. Keep the lowest ratio that holds MMLU inside the tolerance you set, often within one or two points. The two failure modes sit at opposite ends. Too little replay and forgetting continues unchecked. Too much replay and the batches no longer push hard enough on the legal task, so the domain metric stalls.

**Replay** is a small fixed slice of general instruction data interleaved into your domain SFT batches. It pulls the gradient toward the old distribution every step, which slows the drift that produces catastrophic forgetting. The source matters more than people expect. The cleanest choice is the model's own previous SFT corpus, because the distribution already matches what the base was tuned on. When that is not available, a clean public instruction mix like Tulu 3, UltraChat-200k, or OpenAssistant is a reasonable substitute. Pre-tokenise both sources and interleave at the row level so every batch is mixed; concatenating a general block before the domain block does almost nothing because the long domain tail overwrites it. Start at roughly 10% general and 90% domain, then treat the ratio as your main knob. Run short pilots at 5%, 10%, and 15% general, log MMLU and your domain metric on a held-out slice every few hundred steps, and pick the lowest ratio that holds MMLU inside a tolerance you set up front, often within one or two points. The failure modes are symmetric. Under 5% replay is too thin to refresh the general distributions and MMLU keeps falling. Above 20 to 30% the domain signal gets diluted, the target metric plateaus early, and you have quietly converted a domain SFT into a weak continued-pretraining run.