Walk through the triage order when a customer-support LoRA tanks MMLU by 4 points post-deploy

Order steps·Medium·4.0 · 0·~1 min·Relevant atAnthropic Databricks OpenAI

Attempt it

1Roll back if all four levers fail and re-scope the domain dataset for diversity before attempting another training run.
2Lower the learning rate by 2-3x and re-train; smaller steps reduce how far the weights drift from the pretrained distribution.
3Drop the LoRA rank, which limits the magnitude of the additive delta the adapter can apply to the base weights.
4Mix in 5-10% general instruction data (e.g. a sampled slice of the original SFT corpus or a Tulu/UltraChat replay) on the next training run.
5Reduce the number of training epochs and re-evaluate; over-training is the single most common cause of capability regressions on narrow domains.
6Confirm the regression on the MERGED model rather than the adapter-loaded one, to rule out adapter-stack or serving misconfiguration as the source of the drop.

The right order is 0, 1, 2, 3, 4, 5, moving from cheapest verification to most expensive structural change. **Step 0**: confirm the regression on the merged model. Adapter-loaded inference involves a runtime stack that can have its own bugs. Before you assume the model itself regressed, prove it by running the same eval against the merged model with no adapter overhead. If the merged model is clean, the bug is in the serving stack, not the training. **Step 1**: reduce epochs. Over-training is the single most common cause of capability regressions on narrow domains. A model trained for three epochs on a support corpus drifts further from the pretrained distribution than one trained for a single epoch. Drop to one epoch and re-evaluate; the regression often shrinks substantially with no other change. **Step 2**: lower the learning rate by two to three times and re-train. Smaller updates per step keep the weights closer to the pretrained distribution. This is one step more invasive than reducing epochs because it requires a full re-train rather than just early stopping. **Step 3**: mix in 5 to 10 percent general instruction data. Replaying a small slice of the original SFT corpus, or a Tulu / UltraChat replay, anchors the model against forgetting. The model now sees both narrow domain and general reasoning during training, and the gradient never gets fully captured by the support task. **Step 4**: drop the LoRA rank. Lower rank limits the size of the additive delta the adapter can apply, which limits how far the model can drift from the base. This is a structural change to the recipe. **Step 5**: roll back and rescope the dataset. If none of the training levers worked, the support corpus probably lacks the diversity needed to learn the task without forgetting. Rescoping the data is a deeper intervention than any single hyperparameter change.

**The order is 0, 1, 2, 3, 4, 5**, structured from cheapest verification through least invasive training-side levers to a full rollback as the last resort. Step 0 confirms the regression on the merged model. The adapter-loaded inference path has its own bug surface (adapter-stack misconfig, quantization issues, template mismatches). Running the eval on the merged model rules out the serving layer. If merged passes cleanly, the bug is operational and the rest of the order does not apply. Step 1 reduces epochs and is the highest-leverage cheap intervention. Over-training is the single most common cause of regressions on narrow corpora. Each epoch pulls weights further from the pretrained distribution, entirely toward the narrow task. Dropping to one epoch often shrinks the regression substantially with no other change. Step 2 lowers learning rate by 2 to 3 times. Smaller update steps reduce per-step drift, limiting cumulative drift. One step more expensive than step 1 because it requires a full re-train, but still a within-recipe knob. Step 3 mixes in general instruction data at 5 to 10 percent. Replaying a slice of the original SFT corpus or a Tulu / UltraChat replay anchors the model against forgetting. The gradient never gets fully captured by the support task, and general-reasoning cross-entropy opposes drift on MMLU dimensions. Step 4 drops LoRA rank. Lower rank reduces the rank of the additive delta and structurally limits drift. This is a recipe change with knock-on effects on support-task quality, which is why it comes after the data-side intervention in step 3. Step 5 rolls back and rescopes. If the previous four failed, the dataset itself lacks the diversity to learn the support task without sacrificing general capability. Rescoping restarts the project rather than tuning it, which is why it sits last.

Quick reference

TL;DR

Confirm the regression on the merged model first, then sequence training levers from least to most invasive (fewer epochs, lower LR, replay data, smaller rank), and roll back only as a last resort with a dataset rescope.

Memory aid

Easy to grasp

Imagine a star athlete who practised one specific drill so much that they got worse at every other sport they used to play. The right response is not to retire them. First, check whether they really did get worse or if the new uniform is just confusing the referee. Then try the cheap fixes in order: train fewer days a week, train less intensely, mix in cross-training from their old sport, scale back the size of the specialised drill. Only if nothing works do you bench them and re-design the training program from scratch. The athlete is your fine-tuned model. The original sport is general reasoning measured by MMLU. The drill is the support corpus. Each lever in the order shrinks the gap between the new and the old self by a different mechanism.

Key concepts

Catastrophic forgetting LoRA MMLU benchmark Instruction tuning Replay data

Detailed answer & concept explanation

~10 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: walk through 0-5 in order, justify each step's placement by cost vs invasiveness, and explain why rollback comes last despite being the deepest intervention.

Real-world examples

Real products, models, and research that use this idea.

Tulu 3 and UltraChat are widely used general-capability replay corpora, mixed at 5-10 percent into narrow-domain SFT runs to anchor against MMLU-style regression.
Hugging Face TRL and Axolotl both expose checkpoint-level eval hooks that let teams catch general-capability regression mid-run rather than post-deploy.
The original LoRA paper and follow-up work on catastrophic forgetting documented that lower rank reduces drift magnitude, supporting the late-stage placement in the triage order.
Modern fine-tuning recipes for Llama 4 and Qwen 3.5 routinely combine 5 percent general replay with conservative epoch counts to preserve general capability after domain SFT.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you set up CI gates to catch general-capability regression before deployment?

QHow does the choice between Tulu and UltraChat as replay data affect the result?

QWhen would you skip steps 1 through 4 and go straight to rollback?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Jumping straight to rolling back without trying the cheap training-side levers first. Most capability regressions on narrow domains respond to fewer epochs or a lower learning rate within a single retraining cycle.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.