Diagnose and fix an over-refusal regression after a guardrail rollout

Short answer·Medium·4.0 · 0·~3 min·Asked atAnthropic OpenAI Meta

Attempt it

After rolling out a new output toxicity classifier, your refusal rate on benign queries jumped from 1 percent to 8 percent. Walk through how you would diagnose and fix this without simply turning the classifier off.

Free · 2 AI evals / day

The trap with an over-refusal regression is that the obvious fix, lower the classifier threshold, trades one problem for another. You stop refusing benign queries but you also stop blocking genuine harm. The disciplined approach is five steps. **Step 1: quantify.** Sample the newly-refused conversations. Label a few hundred by hand into true refusals (classifier was right) and over-refusals (benign blocked). If true refusals are also up, the classifier may be catching real harm you previously missed; do not conflate that with the over-refusal signal. **Step 2: cluster.** Group the over-refusals into themes. Common patterns: medical-information queries, security-research queries, queries containing certain demographic or topical terms the classifier learned a spurious correlation with. **Step 3: locate the layer.** Which layer actually fired? The input rail, the output classifier, or the LLM's own RLHF refusal? Logs should tell you. Each layer has a different fix. **Step 4: fix at the right layer.** Spurious classifier correlations get fixed by retraining with counter-examples. Policy-correct-but-UX-poor refusals get a better templated message that redirects the user. LLM-level over-refusals usually respond to a system-prompt tweak. **Step 5: lock it in.** Run XSTest (the standard over-refusal benchmark) and add it to CI so future rollouts cannot regress this property without a build failure.

An **over-refusal regression** is a calibration failure, not a binary safety failure. The fix discipline is: quantify, cluster, locate, fix at the right layer, lock in with eval. **Quantify.** Pull a representative sample of newly-refused conversations from the rollout window and the prior window. Label by hand into true refusals (the classifier was right) and over-refusals (benign queries blocked). The split matters because if true refusals are also up, the new classifier is catching real harm you missed before, that signal must not be wiped out by an indiscriminate threshold rollback. **Cluster.** Group the over-refusals to find themes. Common 2026 patterns: - **Medical-information queries** triggering a spurious link between health terms and the classifier's harm taxonomy. - **Security-research queries** triggering offence-related categories. - **Demographic terms** triggering hate-speech categories because of training-set imbalance. - **Code with specific keywords** ("kill", "abort", "hack") triggering violence categories. **Locate.** Inspect logs to identify which layer actually fired. Possible sources: input rail (Llama Prompt Guard, Lakera Guard), output classifier (the one you just rolled out), or the LLM's own RLHF refusal. Each layer has different fix mechanics. **Fix at the right layer.** - Spurious correlations in the classifier: fine-tune with counter-examples or use a managed classifier with category-level threshold tuning. - Policy-correct refusals with bad UX (medical info refused for liability reasons): adjust the templated refusal to redirect ("I cannot give medical advice, but here is how to find a clinician"). - LLM-level over-refusal: a system-prompt tweak or a refusal-suppression eval-based fine-tune. **Lock in.** Run XSTest, the standard over-refusal benchmark (250 benign queries that look superficially unsafe). Add it to CI so any future rollout that pushes over-refusal back up fails the build.

Quick reference

TL;DR

Quantify newly-refused traffic, cluster over-refusals to find themes, identify which layer fires, fix at the lowest-cost layer, and add an XSTest-style benchmark to CI to catch regressions.

Memory aid

Easy to grasp

Imagine a new security guard at your office. On day one, the number of people turned away at the lobby jumps from 1 to 8 percent. You do not fire the guard or remove the metal detector. You ask: who got turned away? You watch the camera, list the names, look for patterns. Maybe everyone wearing red got blocked because the guard misremembered a uniform. Maybe legitimate visitors with crutches set off the metal detector. The fix is different for each cause. Same with a too-strict classifier: you find the pattern, fix the right knob, and add a daily test that prevents the same regression next time.

Key concepts

Safety and Guardrails LLM Evaluation

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 minutes: the quantify cluster locate fix lock in discipline, why over-refusal is a separate metric from safety, how to attribute the firing layer, and how XSTest belongs in CI.

Real-world examples

Real products, models, and research that use this idea.

XSTest (Röttger et al.) is the standard over-refusal benchmark used by Anthropic, OpenAI, and Meta in their model evaluations
Patronus AI's continuous-evaluation product ships XSTest-aligned suites with per-category over-refusal tracking against deployed Claude Opus 4.7 and GPT-5.5 endpoints
Anthropic's Claude system-prompt docs include guidance for reducing over-refusal via system-prompt phrasing rather than classifier changes
Meta Llama Guard 4 ships with per-category threshold settings specifically so refusal calibration can be tuned without changing the underlying classifier

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat if the over-refusal cluster turns out to be on demographic terms, how do you communicate the fix to a governance review?

QHow would you instrument production so the next over-refusal regression is caught in hours, not weeks?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reaching for the classifier threshold knob before any labelling is done, which trades over-refusal for under-refusal without learning what actually broke.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.

Röttger et al., XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours