Describe what OpenAI's Moderation API classifies and what it does not

Flashcard·Easy·4.0 · 0·~30s·Asked atAdobe Canva Fireworks Ai

Attempt it

It is a content-safety classifier returning category scores (harassment, hate, self-harm, sexual, violence, illicit) plus per-category booleans. It is free to use against OpenAI's policy taxonomy. It does NOT detect prompt injection, jailbreak attempts, PII, secrets, off-topic content, or custom per-deployment policy. Treat it as one layer in the input/output rail for OpenAI-policy content categories; pair it with Prompt Guard or Lakera for injection detection and with regex/PII tools for the rest.

Your answer

The OpenAI Moderation API is a free endpoint that takes text (and as of 2024-2025, also images for the omni-moderation model) and returns scores for a set of harm categories: harassment, hate, self-harm, sexual content, violence, illicit, and a few sub-categories. For each one you get a numerical score plus a boolean for whether the text crosses OpenAI's threshold. It is the right tool when you need a quick baseline for the kinds of content OpenAI itself would refuse to produce. You can run it on user input before the LLM sees it and on model output before you return it to a user. What it does not do is just as important. It does not detect prompt injection, strings like ignore previous instructions are not in its taxonomy. It does not redact PII or secrets. It does not know what your particular application considers off-topic or off-brand. It does not enforce your custom policy. For those, you pair it with a separate injection classifier (Prompt Guard, Lakera, Rebuff), a PII scanner, and your own classifier or rules for application-specific risks. The Moderation API is one tile in the safety wall, not the whole wall.

The Moderation API is OpenAI's **content-category classifier**, exposed as a free API endpoint. It returns: - A numerical score per category, harassment, hate, self-harm, sexual content, violence, illicit, plus a few graphic or targeted sub-categories. - A boolean per category indicating whether the score exceeded OpenAI's threshold. - A composite flagged boolean. The text moderation latest and omni-moderation models cover the same taxonomy with the omni variant adding image inputs and improving multilingual coverage. What the API is for: a cheap input or output rail on the specific content categories OpenAI's usage policy already covers. Running every user message and every model output through it gives you a baseline rejection of obvious policy violations, with the latency and cost of a small classifier. What it is **not** for: anything outside that taxonomy. Specifically, it does not: - Detect prompt injection or jailbreak attempts. Strings like ignore the system prompt or do anything now are not flagged. - Recognise PII, secrets, credentials, or financial data. - Understand off-topic content, off-brand language, or your custom policy. - Score factual accuracy or hallucination. Production stacks pair it with a dedicated injection rail (Llama Guard 4, Prompt Guard, Lakera Guard, Rebuff), a PII recogniser (Microsoft Presidio, AWS Comprehend, or a regex layer for known formats), and a policy classifier you control for application-specific risks. The Moderation API is one tile; the safety wall is everything together.

TL;DR

OpenAI's Moderation API is a free content-category classifier over OpenAI's policy taxonomy; it does not detect prompt injection, PII, secrets, off-topic content, or your custom policy.

Memory aid

Easy to grasp

Imagine the Moderation API as a smoke alarm tuned to one company's idea of dangerous smoke. It will beep loudly when it smells the things on its list, like harassment or violent threats, and it will tell you which kind of smoke it noticed. It does not care about the smell of someone whispering instructions to a robot to misbehave, it does not notice if a phone number has just floated past, and it does not know whether your house has a rule about no candles in the bedroom. The alarm is genuinely useful for what it covers, but it covers a narrow set of smells. You still need other alarms for everything else, and you have to install them yourself.

Key concepts

Safety and guardrails

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The OpenAI Moderation API has been the default first stop for content safety in OpenAI-backed applications since GPT-3 launched. It is free, it is fast, and it is well-aligned with OpenAI's own model behaviour. Those properties also encourage a recurring mistake: treating it as the general-purpose safety layer for the application.

This card describes what the API actually does, then walks through three categories it explicitly does not cover, then sketches the layered architecture that uses it correctly. The aim is to make you fluent enough to defend the layer's role to a skeptical interviewer or to a security review.

What the Moderation API actually returns

The Moderation API is a classifier with a well-defined output shape. Send it text (and, in the omni-moderation model, optionally an image), receive back:

A per-category numerical score from 0 to 1 across categories like harassment, hate, self-harm, sexual content, violence, illicit, plus a few sub-categories such as harassment/threatening and self-harm/intent.
A per-category boolean flag indicating whether the score crossed OpenAI's threshold.
A top-level flagged boolean that is the disjunction of the per-category flags.

Why both scores and flags

The flags are a convenience for the simple case of refuse if any category tripped. The scores are the operating-point control. A consumer-product team might decide that any score above 0.3 on harassment is unacceptable for their family-friendly audience; an enterprise legal tool might accept a higher threshold because the cost of false positives is also high. The scores let you tune.

Cost and latency

The API is free at time of writing; the omni-moderation model adds image input and improved multilingual coverage at the same price point. Latency is in the tens of milliseconds, comparable to a small embedding call, so it fits inline in both input and output rails.

Three threats the Moderation API does not detect

How to wire it into the rails correctly

Threshold calibration and drift monitoring

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's omni moderation latest (released late 2024) added image input and improved multilingual coverage; teams running GPT-5.5 chat experiences typically still pair it with a separate injection rail.
Lakera Guard and Microsoft Presidio are commonly stacked alongside the Moderation API in production OpenAI deployments precisely because the Moderation taxonomy does not cover injection or PII.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you choose the per-category threshold to use instead of the default boolean?

Build a labelled adversarial dataset for your application, sweep thresholds, and pick the operating point where your false-positive cost equals your false-negative cost.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating Moderation API as a general-purpose safety gate; it is a content-category classifier and misses injection, PII, secrets, off-topic, and any custom policy you care about.

60 second bullets to scan on the way to the call.

What categories the Moderation API returns
Score vector versus boolean flags and how to choose thresholds

Primary sources. Browse if you want the original framing.