Why run a toxicity classifier on output even when the model was RLHF tuned to refuse harmful content

Flashcard·Easy·4.0 · 0·~30s·Asked atGraphcore Neo4j

Attempt it

RLHF reduces the base rate on benign distributions but does not generalize to adversarial inputs or to indirect injection that arrives in retrieved content. A dedicated output classifier acts as a deterministic backstop: it inspects the actual emitted tokens against an auditable policy taxonomy, fires on cases the model itself was tricked into producing, and creates an evidence trail your incident review can replay. It also enforces per-deployment policy the base model never trained on (e.g. competitor-name suppression).

Your answer

RLHF training makes the model less likely to produce harmful output on a typical user input. That is a reduction in base rate, not a guarantee. Several gaps remain that justify a separate output-side classifier. **Adversarial input.** RLHF was trained on a particular distribution of inputs and refusals. Jailbreaks and persuasion attacks specifically explore the corners of the input space the training did not cover. When one succeeds, the model emits content that violates the policy it was trained to refuse. A separate classifier inspecting the actual output tokens catches the result regardless of how the input was framed. **Indirect injection.** Content the model retrieves, webpages, documents, search results, can contain instructions the model picks up. RLHF on the model's behaviour does not cover the case where the retrieved content told the model to produce a harmful output. **Per-deployment policy.** Your product has policy the base model never trained on. Maybe you suppress competitor names, maybe you enforce a regulatory disclaimer, maybe you cannot mention specific medical conditions. The base model has no idea about those rules; the deployment-specific classifier enforces them. **Audit trail.** RLHF gives you a probability the model refuses. A classifier gives you a verdict on each call with a logged score, which is what incident review and compliance need.

**RLHF is a soft prior, not a control.** It lowers the rate of harmful output on the distribution it was trained on. It does not eliminate harmful output and does not generalise cleanly outside that distribution. A separate output-side toxicity classifier adds four things RLHF cannot provide. **Adversarial coverage**: when a jailbreak or persuasion attack succeeds and the model emits content it would normally refuse, the classifier catches the emission regardless of how the model was tricked. **Indirect-injection coverage**: when retrieved content told the model to produce harmful output, the classifier inspects the output independently of the input framing. **Per-deployment policy**: the classifier enforces rules the base model never trained on, competitor suppression, regulated-industry restrictions, regional content policies. **Audit trail**: the classifier produces a deterministic verdict with a logged score, which is what incident review, regulatory compliance, and red-team replay need. The two layers are complementary. RLHF reduces the rate at which the classifier needs to fire. The classifier handles the cases RLHF missed and the policies RLHF could not encode. Removing either makes the system materially weaker; relying on either alone is a misunderstanding of what each provides.

TL;DR

RLHF is a soft prior that fails under adversarial input and cannot encode per-deployment policy; a separate output classifier is the deterministic backstop with an audit trail.

Memory aid

Easy to grasp

Imagine a restaurant where the chef has been trained for years not to serve raw chicken. Most days, the food is fine. But people sometimes still get sick, because the chef has a bad day, because a supplier slipped in contaminated stock, or because someone deliberately tricked the chef into a shortcut. A health inspector at the kitchen door tasting every plate before it goes out catches the cases the chef missed. The inspector does not replace the chef's training; it backs the training up with an independent check that has nothing invested in the chef being right.

Key concepts

Safety and Guardrails RLHF

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

RLHF-tuned models refuse most harmful requests most of the time. That is a real safety contribution. It is also not a complete safety system, and treating it as one is a mistake organisations make routinely.

The gap between 'usually refuses' and 'will never emit unsafe output' is where the output classifier lives. Knowing exactly what the classifier adds, and why it cannot be replaced by stronger model training, is the difference between a safety stack that works under adversarial pressure and one that looks good on a slide.

What RLHF training actually does, and does not, give you

RLHF takes a base language model and tunes it against a reward signal that incorporates human preferences. Part of the preference signal is 'refuse harmful requests'. The result is a model whose probability distribution puts much less mass on harmful responses to typical inputs.

What that gives you is a lower base rate. Inputs that look like the training distribution produce safer outputs than they would from an un-tuned model. On benign user interactions, the harmful-output rate is very small.

What it does not give you is a guarantee. The model is still a probability distribution over outputs. Inputs that look unlike the training distribution, jailbreaks, persuasion attacks, indirect injection via retrieved content, push the model into regions where the safety prior is weaker. The model can and does emit harmful output in those regions. The question is whether a second, independent layer is watching when it happens.

Gap one: adversarial input

Gap two: indirect injection through retrieval

Gap three: per-deployment policy the base model never trained on

Audit, compliance, and the case for an explicit verdict

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Meta Llama Guard 4 (2026), open-weights multimodal output classifier with configurable taxonomy, deployed alongside Llama 4 generation models.
OpenAI Moderation API, hosted text and image classifier paired with GPT models for output-side filtering.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you tune the classifier threshold for a customer-support bot vs a code-generation bot?

Talk about the cost asymmetry of false positives (annoying refusal) vs false negatives (harmful emission), the different harm taxonomies for the two surfaces, and using a paired benign-suite Pareto plot to pick the operating point.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing RLHF-tuned refusals are a sufficient safety control. They reduce the base rate; they do not generalise to adversarial input or to per-deployment policy.

60 second bullets to scan on the way to the call.

Explain why RLHF is a probability distribution rather than a deterministic refusal
Name three categories of harmful output the classifier catches that RLHF misses

Primary sources. Browse if you want the original framing.