Walk through the triage order when a customer support LoRA tanks MMLU by 4 points post-deploy

Order steps·Medium·4.0 · 0·~1 min·Asked atHebbia Ltimindtree Turing·Relevant atAnthropic Databricks OpenAI

Attempt it

1Roll back if all four levers fail and re-scope the domain dataset for diversity before attempting another training run.
2Lower the learning rate by 2-3x and re-train; smaller steps reduce how far the weights drift from the pretrained distribution.
3Drop the LoRA rank, which limits the magnitude of the additive delta the adapter can apply to the base weights.
4Mix in 5-10% general instruction data (e.g. a sampled slice of the original SFT corpus or a Tulu/UltraChat replay) on the next training run.
5Reduce the number of training epochs and re-evaluate; over training is the single most common cause of capability regressions on narrow domains.
6Confirm the regression on the MERGED model rather than the adapter loaded one, to rule out adapter stack or serving misconfiguration as the source of the drop.

The right order is 0, 1, 2, 3, 4, 5, moving from cheapest verification to most expensive structural change. **Step 0**: confirm the regression on the merged model. Adapter-loaded inference involves a runtime stack that can have its own bugs. Before you assume the model itself regressed, prove it by running the same eval against the merged model with no adapter overhead. If the merged model is clean, the bug is in the serving stack, not the training. **Step 1**: reduce epochs. Over-training is the single most common cause of capability regressions on narrow domains. A model trained for three epochs on a support corpus drifts further from the pretrained distribution than one trained for a single epoch. Drop to one epoch and re-evaluate; the regression often shrinks substantially with no other change. **Step 2**: lower the learning rate by two to three times and re-train. Smaller updates per step keep the weights closer to the pretrained distribution. This is one step more invasive than reducing epochs because it requires a full re-train rather than just early stopping. **Step 3**: mix in 5 to 10 percent general instruction data. Replaying a small slice of the original SFT corpus, or a Tulu / UltraChat replay, anchors the model against forgetting. The model now sees both narrow domain and general reasoning during training, and the gradient never gets fully captured by the support task. **Step 4**: drop the LoRA rank. Lower rank limits the size of the additive delta the adapter can apply, which limits how far the model can drift from the base. This is a structural change to the recipe. **Step 5**: roll back and rescope the dataset. If none of the training levers worked, the support corpus probably lacks the diversity needed to learn the task without forgetting. Rescoping the data is a deeper intervention than any single hyperparameter change.

**The order is 0, 1, 2, 3, 4, 5**, structured from cheapest verification through least invasive training-side levers to a full rollback as the last resort. Step 0 confirms the regression on the merged model. The adapter-loaded inference path has its own bug surface (adapter-stack misconfig, quantization issues, template mismatches). Running the eval on the merged model rules out the serving layer. If merged passes cleanly, the bug is operational and the rest of the order does not apply. Step 1 reduces epochs and is the highest-leverage cheap intervention. Over-training is the single most common cause of regressions on narrow corpora. Each epoch pulls weights further from the pretrained distribution, entirely toward the narrow task. Dropping to one epoch often shrinks the regression substantially with no other change. Step 2 lowers learning rate by 2 to 3 times. Smaller update steps reduce per-step drift, limiting cumulative drift. One step more expensive than step 1 because it requires a full re-train, but still a within-recipe knob. Step 3 mixes in general instruction data at 5 to 10 percent. Replaying a slice of the original SFT corpus or a Tulu / UltraChat replay anchors the model against forgetting. The gradient never gets fully captured by the support task, and general-reasoning cross-entropy opposes drift on MMLU dimensions. Step 4 drops LoRA rank. Lower rank reduces the rank of the additive delta and structurally limits drift. This is a recipe change with knock-on effects on support-task quality, which is why it comes after the data-side intervention in step 3. Step 5 rolls back and rescopes. If the previous four failed, the dataset itself lacks the diversity to learn the support task without sacrificing general capability. Rescoping restarts the project rather than tuning it, which is why it sits last.

TL;DR

Confirm the regression on the merged model first, then sequence training levers from least to most invasive (fewer epochs, lower LR, replay data, smaller rank), and roll back only as a last resort with a dataset rescope.

Memory aid

Easy to grasp

Imagine a star athlete who practised one specific drill so much that they got worse at every other sport they used to play. The right response is not to retire them. First, check whether they really did get worse or if the new uniform is just confusing the referee. Then try the cheap fixes in order: train fewer days a week, train less intensely, mix in cross-training from their old sport, scale back the size of the specialised drill. Only if nothing works do you bench them and re-design the training program from scratch. The athlete is your fine-tuned model. The original sport is general reasoning measured by MMLU. The drill is the support corpus. Each lever in the order shrinks the gap between the new and the old self by a different mechanism.

Key concepts

Catastrophic forgetting LoRA MMLU benchmark Instruction tuning Replay data

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Capability regression after narrow-domain fine-tuning is one of the most common operational issues in production LLM teams. A LoRA fine-tune lifts the target task metric, deploys cleanly, and a week later the on-call rotation discovers that MMLU or another general-capability benchmark has dropped several points. The team now has a working model on the narrow task and a regressed model on the general one. The question is what to do, and in what order.

The right answer is not 'try everything at once'. Each lever has a different cost and a different scope. Some levers (epoch reduction, learning rate adjustment) cost a single retraining cycle and stay within the original recipe. Others (replay data, rank changes) require structural changes to the recipe and reset some assumptions of the original training plan. The deepest lever (rollback and dataset rescope) effectively restarts the project. A triage process that respects the cost gradient resolves most regressions in the cheapest steps and saves the deepest intervention for cases where it is genuinely needed.

This question puts the six standard levers in front of the candidate and asks for the right order. The correct sequencing is 0, 1, 2, 3, 4, 5, moving from a verification step that costs nothing through within-recipe knobs to structural changes and finally to a rollback. The principles are: verify before you intervene, and apply cheaper interventions before structural ones.

Step 0: verify the regression on the merged model

The first step in any post-deployment regression is to confirm the regression is real and attributable to the trained weights. Adapter-loaded inference runs through PEFT's dynamic injection of LoRA matrices into base modules, a serving stack that handles quantization and template assembly, and any optimisations the inference framework applies on top. Each of those layers has its own potential for bugs that can produce a regression that has nothing to do with the training itself.

The verification step is straightforward. Take the trained adapter, merge it into the base model to produce a single weight tensor with no adapter overhead, run the merged model through the same eval that flagged the regression, and compare the score. If the merged model passes the eval cleanly, the regression is operational and lives in the adapter-loaded inference path. The investigation moves to the serving stack and the rest of this triage order does not apply.

If the merged model also shows the regression, the regression is in the trained weights themselves and the rest of the order applies. The verification step takes minutes to run and saves potentially days of training experiments chasing a bug that lives in serving configuration. Production teams that skip this step routinely discover, after a week of retraining, that the bug was a template mismatch in the inference stack that an hour of debugging would have fixed.

The step is at position zero rather than as part of the main triage because it is verification, not intervention. It does not change anything about the trained model; it tells you whether the rest of the triage order should run.

Step 1: reduce epochs as the highest-leverage cheap knob

Step 2: lower the learning rate

Step 3: replay general instruction data

Step 4: drop the LoRA rank

Step 5: rollback and dataset rescope

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Tulu 3 and UltraChat are widely used general-capability replay corpora, mixed at 5-10 percent into narrow-domain SFT runs to anchor against MMLU-style regression.
Hugging Face TRL and Axolotl both expose checkpoint-level eval hooks that let teams catch general-capability regression mid-run rather than post-deploy.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you set up CI gates to catch general-capability regression before deployment?

Add an MMLU or equivalent eval to the training loop at every checkpoint. Set a threshold (e.g. no more than 1 point below base) that blocks checkpoint promotion. The gate moves the regression catch from post-deploy to mid-training and reduces triage scope by an order of magnitude.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Jumping straight to rolling back without trying the cheap training-side levers first. Most capability regressions on narrow domains respond to fewer epochs or a lower learning rate within a single retraining cycle.

60 second bullets to scan on the way to the call.

Why merged-model verification rules out serving-layer bugs before training experiments
Why epoch count is the largest single driver of post-FT capability regression

Primary sources. Browse if you want the original framing.