A consumer-facing LLM product is about to launch. Describe the safety evaluation that should run first. · Zenaique

Attempt it

A consumer-facing LLM product is about to launch. Describe the safety evaluation that should run first.

Free · 2 AI evals / day

A consumer LLM launch needs a safety evaluation that covers four areas: harmful content, PII leakage, jailbreak resistance, and demographic bias. The evaluation uses red-teaming: sending adversarial prompts to the model and checking whether it refuses. Manual red-teaming uses skilled human adversaries who probe creatively. Automated red-teaming uses adversarial prompt generators to cover thousands of attack patterns. You need both: manual catches novel attacks, automated catches systematic weaknesses. The metric that matters most is the balance between two rates. Refusal rate on adversarial prompts measures how often the model correctly refuses harmful requests. False-refusal rate on benign prompts measures how often the model incorrectly refuses legitimate questions. A model that refuses everything scores perfectly on adversarial refusal rate but is useless. A model that refuses nothing scores perfectly on false-refusal rate but is dangerous. The launch decision is about finding the operating point where adversarial refusal is above a threshold (e.g. 95%) and false-refusal is below a threshold (e.g. 2%). Beyond content, test for PII leakage (extraction attacks on training data) and demographic bias (does the model refuse more questions about certain groups than others?).

**Pre-launch safety eval** has four pillars. First, red-teaming across harm categories: harmful content generation, PII leakage, jailbreak resistance (prompt injection, role-play exploits, encoding tricks), and demographic bias. Use manual red-teamers for creative novel attacks and automated adversarial prompt generators for systematic coverage at scale. Second, refusal calibration. Build a test set with adversarial prompts across every harm category paired with topically adjacent benign prompts (e.g. a benign medical question one word from a harmful self-harm query). Measure refusal rate on adversarial prompts and false-refusal rate on benign prompts. Third, PII and data leakage testing. Probe with extraction attacks (membership inference, verbatim training-data extraction) and context-window exfiltration via prompt injection. Fourth, bias audit. Measure output quality and refusal rate across demographic groups. Differential refusal across demographics is a bias signal even when the content itself is safe. The metric that matters most is the refusal rate / false refusal rate balance. A model that refuses everything is safe but unusable. One that refuses nothing is dangerous. The launch threshold is adversarial refusal above 95% and false-refusal below 2%. This pair defines the operating point, and every other safety metric feeds into calibrating it.

Quick reference

TL;DR

Run manual plus automated red-teaming across harm categories, PII leakage, jailbreaks, and bias. The metric that matters most is the balance between adversarial refusal rate and benign false-refusal rate.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you are hiring a security guard for a store. You test two things: does the guard stop real shoplifters (refusal rate), and does the guard accidentally block paying customers (false-refusal rate)? A guard who blocks everyone is safe but drives away business. A guard who blocks no one lets shoplifters walk out. The best guard catches the thieves and lets the customers through. A safety eval for an LLM works the same way: you send it trick questions to see if it refuses, and you send it normal questions to make sure it still helps.

Key concepts

LLM Evaluation AI Safety

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Lead with the precision-recall framing: adversarial refusal is recall, benign helpfulness is precision. Walk the four pillars: red-teaming (manual plus automated), refusal calibration (paired adversarial and benign test sets), PII leakage (extraction attacks), and bias audit (differential refusal). Name the metric pair that dominates the launch decision. Close with the evolving threat landscape and why quarterly refreshes are necessary.

Real-world examples

Real products, models, and research that use this idea.

Anthropic publishes safety evaluations for Claude that measure adversarial refusal rate across harm categories and report false-refusal rates on benign test sets as a key usability indicator.
OpenAI's red-teaming for GPT-5.5 and o-series models combines external manual red-teamers with automated adversarial prompt generators to cover both novel attacks and systematic patterns.
Google DeepMind's Gemini safety evaluations include demographic bias audits that measure differential refusal rates across demographic groups as a fairness metric.
The OWASP LLM Top 10 lists prompt injection and training-data extraction as top risks for consumer LLM products, mapping directly to the jailbreak resistance and PII leakage pillars.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour model passes safety eval with 97% adversarial refusal and 1.5% false-refusal. A week after launch, a novel jailbreak goes viral on social media. What failed and what do you do?

Sign in or go Pro to see the approach for this follow-up.

QYour bias audit shows that the model refuses 8% of questions mentioning one demographic group but only 2% for another. Both rates are below the absolute false-refusal threshold. Is this a problem?

Sign in or go Pro to see the approach for this follow-up.

QHow do you maintain the safety evaluation after launch as the model and the threat landscape evolve?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Optimizing only for adversarial refusal rate and shipping a model that refuses too many benign requests, making the product frustrating to use for legitimate queries.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.