Why run a toxicity classifier on output even when the model was RLHF-tuned to refuse harmful content

Attempt it

RLHF reduces the base rate on benign distributions but does not generalize to adversarial inputs or to indirect injection that arrives in retrieved content. A dedicated output classifier acts as a deterministic backstop: it inspects the actual emitted tokens against an auditable policy taxonomy, fires on cases the model itself was tricked into producing, and creates an evidence trail your incident review can replay. It also enforces per-deployment policy the base model never trained on (e.g. competitor-name suppression).

Your answer

RLHF training makes the model less likely to produce harmful output on a typical user input. That is a reduction in base rate, not a guarantee. Several gaps remain that justify a separate output-side classifier. **Adversarial input.** RLHF was trained on a particular distribution of inputs and refusals. Jailbreaks and persuasion attacks specifically explore the corners of the input space the training did not cover. When one succeeds, the model emits content that violates the policy it was trained to refuse. A separate classifier inspecting the actual output tokens catches the result regardless of how the input was framed. **Indirect injection.** Content the model retrieves, webpages, documents, search results, can contain instructions the model picks up. RLHF on the model's behaviour does not cover the case where the retrieved content told the model to produce a harmful output. **Per-deployment policy.** Your product has policy the base model never trained on. Maybe you suppress competitor names, maybe you enforce a regulatory disclaimer, maybe you cannot mention specific medical conditions. The base model has no idea about those rules; the deployment-specific classifier enforces them. **Audit trail.** RLHF gives you a probability the model refuses. A classifier gives you a verdict on each call with a logged score, which is what incident review and compliance need.

**RLHF is a soft prior, not a control.** It lowers the rate of harmful output on the distribution it was trained on. It does not eliminate harmful output and does not generalise cleanly outside that distribution. A separate output-side toxicity classifier adds four things RLHF cannot provide. **Adversarial coverage**: when a jailbreak or persuasion attack succeeds and the model emits content it would normally refuse, the classifier catches the emission regardless of how the model was tricked. **Indirect-injection coverage**: when retrieved content told the model to produce harmful output, the classifier inspects the output independently of the input framing. **Per-deployment policy**: the classifier enforces rules the base model never trained on, competitor suppression, regulated-industry restrictions, regional content policies. **Audit trail**: the classifier produces a deterministic verdict with a logged score, which is what incident review, regulatory compliance, and red-team replay need. The two layers are complementary. RLHF reduces the rate at which the classifier needs to fire. The classifier handles the cases RLHF missed and the policies RLHF could not encode. Removing either makes the system materially weaker; relying on either alone is a misunderstanding of what each provides.

Quick reference

TL;DR

RLHF is a soft prior that fails under adversarial input and cannot encode per-deployment policy; a separate output classifier is the deterministic backstop with an audit trail.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a restaurant where the chef has been trained for years not to serve raw chicken. Most days, the food is fine. But people sometimes still get sick, because the chef has a bad day, because a supplier slipped in contaminated stock, or because someone deliberately tricked the chef into a shortcut. A health inspector at the kitchen door tasting every plate before it goes out catches the cases the chef missed. The inspector does not replace the chef's training; it backs the training up with an independent check that has nothing invested in the chef being right.

Key concepts

Safety and Guardrails RLHF

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

7 min: position RLHF as a soft prior, walk the three coverage gaps (adversarial, indirect, per-deployment), set up the audit-trail argument, and close on calibration discipline.

Real-world examples

Real products, models, and research that use this idea.

Meta Llama Guard 4 (2026), open-weights multimodal output classifier with configurable taxonomy, deployed alongside Llama 4 generation models.
OpenAI Moderation API, hosted text and image classifier paired with GPT models for output-side filtering.
AWS Bedrock Guardrails (2026), per-deployment policy enforcement including PII redaction, denied topics, and contextual grounding checks on top of the base model.
Anthropic's content-safety filters on Claude, separate layer from the model's RLHF training, running on input and output.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you tune the classifier threshold for a customer-support bot vs a code-generation bot?

Sign in or go Pro to see the approach for this follow-up.

QWhat if the classifier and the main model disagree on whether the output is harmful?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing RLHF-tuned refusals are a sufficient safety control. They reduce the base rate; they do not generalise to adversarial input or to per-deployment policy.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.