Describe how you would design an automated red teaming pipeline for a production LLM assistant. What are the key components, how do you measure success, and what human oversight is needed?
A safety red-team program is a measurement loop: an attacker generates diverse adversarial prompts, the target answers, a judge scores harm by category and severity, and humans curate seeds plus validate the judge.
Think of testing a new bank vault before it opens. You hire ethical lockpickers whose only job is to try every trick to break in, while a security expert watches each attempt and writes down which ones got close. You do not test once and declare victory. You keep testing forever, because thieves invent new tricks every month. The lockpickers are clever but blind to attacks they were never taught, so the head of security also dreams up fresh angles by hand. Every break-in attempt gets scored by how bad it would be: a scratch on the paint matters less than an open door. You weight the scary outcomes heavily, fix them first, and re-run last month's successful break-ins to make sure your patches still hold.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A production safety red-team program is best understood as an instrument for estimating residual risk: given everything an adversarial user might try, how often does the assistant produce a harmful response, broken down by how bad that harm is? The job of the pipeline is to make that estimate broad, severity-aware, and trustworthy enough to gate a release.
The defensive framing matters. We generate adversarial prompts only to find and close vulnerabilities before real users or malicious actors exploit them. Everything below assumes that purpose: harm discovery feeding directly into mitigation and regression.
The design has four pillars. An attack taxonomy that defines coverage, an automated attacker, target, and judge loop that scales breadth, a severity-weighted scoring scheme that ranks what matters, and a continuous human in the loop process that compensates for the automation's blind spots and validates the judge.
Start with an attack taxonomy, not a prompt list
The first design decision is what counts as harm and how the space is partitioned. Coverage is measured against an explicit taxonomy, not against a raw count of prompts. A typical taxonomy spans harmful instructions, PII extraction, jailbreak and prompt injection, bias and stereotyping, misinformation, self-harm content, and uplift on dangerous specialised knowledge.
Why this comes first: ten thousand prompts that all probe one category give a falsely reassuring sense of thoroughness while leaving entire harm classes untouched. The taxonomy turns coverage into something auditable. For each category you can ask how many distinct attack strategies were tried, how many prompts, and what the current success rate is.
The taxonomy is also the contract with stakeholders. Safety, legal, and policy teams agree on the categories and on relative severity up front. Without that agreement the downstream scores are uninterpretable, because there is no shared definition of what the pipeline is even trying to prevent.
A mature taxonomy is hierarchical. Top-level categories split into concrete sub-attacks: under jailbreak you list role-play framing, hypothetical framing, encoded payloads, many-shot priming, and prompt injection from retrieved content. This second level is where coverage gaps actually hide, so the report tracks success rate per sub-attack, not just per top-level category. The taxonomy is a living document that grows every time human red-teamers or external researchers discover a new attack class, and each addition becomes a new coverage requirement the automated layer must satisfy.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic and OpenAI run dedicated red-team programs combining human red-teamers with automated attack generation before frontier releases like Claude Opus 4.7 and GPT-5.5.
- Microsoft PyRIT is an open framework for automated LLM red-teaming with attacker, target, and scoring orchestration.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle the attacker LLM's blind spots so coverage does not silently collapse?
Inject human-curated seeds for under-covered categories, use less-aligned or fine-tuned attacker variants, add gradient-guided suffix search, and audit per-category prompt counts so no category starves.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating red-teaming as a one-shot pre-launch checklist instead of a continuous regression suite. New jailbreaks appear weekly, so a single passing run guarantees nothing about next month.
60 second bullets to scan on the way to the call.
The three pipeline roles: attacker, target, and judge classifier
Why attack diversity matters more than raw prompt volume
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.