Naive safety FT failed: model now refuses 'how does anesthesia work?'. Best fix?

Safety fine-tuning teaches a model to refuse harmful requests. When it is done carelessly, the model learns the wrong lesson: it refuses anything that LOOKS dangerous, even when the request is harmless. That is called **over-refusal**. The anesthesia case is classic. A medical student asks how anesthesia works, and the model refuses because the topic touches drugs and unconsciousness. The model never learned that a textbook question is fine. It only learned that certain words feel risky. The real cause is the training data. The safety examples were almost all harmful, and they were full of scary keywords. So the model took a shortcut and tied those keywords to refusal. The fix is to add benign hard-negatives. These are safe questions that contain the same scary keywords, paired with genuinely helpful answers. After retraining, the model learns to judge intent, not surface words. You also need an over-refusal evaluation set, like XSTest, to confirm the model stopped blocking harmless questions without becoming unsafe again.

Over-refusal means the model rejects benign requests because aggressive safety fine-tuning taught it to react to keywords, not intent. The anesthesia refusal is the tell. The root cause is data imbalance. The safety set was almost entirely harmful examples, and those examples were dense with risky-sounding words. With no benign counterexamples, the model learned a shortcut: see the keyword, emit the refusal. The correct fix targets the data. You add benign hard-negatives, which are safe queries that surface-match dangerous ones, paired with the actual helpful answer. Medical anatomy, chemistry homework, and fiction involving conflict all qualify. Retraining on this contrastive mix forces an intent boundary instead of a keyword boundary. Threshold tuning at inference does not fix this. It only slides the operating point of a miscalibrated classifier. Removing safety data entirely is worse, since system-prompt rules are bypassable. You must also measure the change. XSTest is the canonical over-refusal benchmark, and you pair it with a genuine harmful-prompt set. The goal is to cut false refusals while holding the true refusal rate steady.