Diagnose an over refusing model in production

Short answer·Hard·4.0 · 0·~3 min·Asked atSamsung Tesla

Attempt it

Support tickets complain your model refuses benign questions like 'how do I delete my account?' or 'help me write a strongly worded email to my landlord.' Show the refusal calibration investigation: where do you look, what metrics, and what knobs do you have?

Free · 2 AI evals / day

Over-refusal in production is rarely uniform. It is concentrated on specific words, intents, or topics. The first move is to slice the refusal-rate dashboard by intent. If 'account deletion' refuses 30% of requests but 'password reset' refuses 1%, the problem is not the model in general. It is whatever layer over-couples 'delete' with harm. Pull a sample of refused traces and locate the failing layer. There are three places refusals come from: - The safety classifier on the input or output rail (Llama Guard, Azure AI Content Safety). Its taxonomy may map 'account deletion' to a self-harm category by mistake. - The system prompt itself, which may say 'refuse anything related to deletion or destruction' too broadly. - The base model's RLHF tuning, which can over-refuse certain word stems. Each has a different fix. Tune the classifier threshold or disable the wrong category for that rail. Rewrite the system-prompt policy more precisely. Or, if you cannot retrain, add system-prompt context that raises the prior toward compliance ('user requests about account operations are in-scope'). Measure the fix on a paired benign and adversarial holdout so you do not trade a false-refusal win for a jailbreak loss.

**Refusal calibration is a two-axis problem.** Step one: do not trust an aggregate refusal-rate dashboard. Slice the rate by intent, using user-stated intent, an intent classifier, or a topic clusterer over recent traffic. The pattern is usually clear: 'account deletion' refuses at 30%, 'password reset' at 1%, 'strongly-worded email' at 20%, 'polite email' at 0.5%. The problem is concentrated, not uniform. **Step two: locate the failing layer.** Pull 20-50 refused traces from each high-refusal intent. Three layers can be the source: - **Safety classifier on the input or output rail.** Llama Guard 4 or an internal classifier may fire on 'delete' under a category like self-harm. Knob: disable the wrong category for that rail, retrain with counterexamples, or raise the threshold. - **System prompt.** A broad policy like 'refuse anything related to deletion' over-fires on benign account operations. Knob: tighten the policy text and add explicit in-scope examples. - **Base model RLHF tuning.** Some base models over-couple specific word stems with harm. Knob: prepend an in-scope clarifier in the system prompt, or escalate to model-vendor support for a tuning revision. **Step three: measure the fix on paired holdouts.** Build a 100-item benign-refused set and a 100-item adversarial-jailbreak set. A real fix lowers the benign-refusal rate without raising jailbreak success. If jailbreak success rises while benign refusals fall, you have lowered the safety bar, not calibrated it.

TL;DR

Slice refusal rate by intent first, isolate the failing layer (classifier, system prompt, or base model), tune the knob, then validate on paired benign and adversarial sets.

Memory aid

Easy to grasp

Imagine a coffee shop where the barista refuses to make any drink with the word 'shot' in it because 'shot' could mean something dangerous. Customers asking for an espresso shot get turned away while customers ordering a latte are fine. The shop owner does not need to retrain the barista from scratch. They need to slice the complaints by drink, find that the word 'shot' is the trigger, and update the rule. Refusal calibration in a production LLM is the same workflow: slice the data, find the trigger, fix the layer responsible, and verify nothing else broke.

Key concepts

Safety & Guardrails LLMOps LLM Evaluation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Over-refusal is a common safety regression in production LLM deployments. It rarely arrives as a uniform shift. It arrives concentrated on specific intents, word stems, or surfaces. A team that treats the dashboard's global refusal-rate number as the signal will spend weeks adjusting global thresholds without finding the source. A team that slices by intent finds the root cause in an afternoon.

The calibration workflow has four steps: slice the rate by intent, locate the failing layer, tune the right knob, and validate on a paired benign and adversarial holdout. It scales from a single ticket to a full release-gate audit.

Step 1: slice refusal rate by intent

Aggregate refusal-rate dashboards are misleading. A global 3% number can hide intents at 30% and intents at 0.5%. The first move is always to bucket recent refused traces by intent.

Three practical approaches:

Intent classifier on user messages. A small classifier (often a fine-tuned BERT or sentence embedding + cluster) labels each request. Refusal rate is computed per label.
Topic clustering on recent traffic. Embed user messages, cluster, label the top clusters by hand, and bucket the dashboard.
Template matching for known surfaces. For product surfaces with known request shapes (account operations, password flows, billing questions), exact-match templates are enough.

Whatever the method, the goal is the same: produce a refusal-rate column indexed by intent. The pattern usually shows concentration: a small share of intents drives most of the over-refusal complaints. That concentration is where to start.

Step 2: locate the failing layer

Step 3: tune the right knob

Step 4: validate on paired holdouts

Operational discipline and edge cases

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's o1 system card reports over-refusal on benign prompts (including XSTest) next to its disallowed-content results
JailbreakBench pairs 100 harmful behaviors with 100 matched benign behaviors, so attack success and over-refusal are measured together

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build the intent classifier or clusterer that drives the per-intent slicing?

Cluster recent user messages with an embedding model and review the top clusters with a human. Tag clusters with intent labels. Re-use the same labels to slice refusal-rate dashboards and to bucket the holdout set. Keep the labels updated as product surfaces change.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Looking at aggregate refusal rate, declaring it 'looks fine,' and missing that specific intents like 'delete' or 'strongly-worded' are over-refused at 20-30x the baseline.

60 second bullets to scan on the way to the call.

Slice refusal rate by intent before any other diagnosis
Three or four layers a refusal can come from

Primary sources. Browse if you want the original framing.