**Red-teaming** is the practice of deliberately attacking your own model to find where it breaks. In safety evaluation that means crafting prompts designed to make the model produce harmful, toxic, or policy-violating output, so you can find and fix the weakness before a real user does.
The word comes from military and security drills, where a 'red team' plays the attacker against the defending 'blue team'. For an LLM, the attacker writes adversarial prompts: jailbreak attempts, role-play tricks, encoded requests, and multi-turn manipulation. Each prompt that succeeds reveals a hole in the model's safety training.
Manual red-teaming, where humans hand-craft attacks, is thorough and creative but slow and expensive. You cannot pay people to write a million attack prompts.
**Automated red-teaming** fixes the scale problem. You point a second 'attacker' LLM at your target model, have it generate thousands of adversarial prompts automatically, then measure how often the target gives an unsafe answer.
The catch is the correct answer's warning: the attacker LLM only knows attacks inside its own training distribution. It systematically misses whole categories of attack, so a clean automated pass can give false confidence rather than real safety.
**Red-teaming** is adversarial safety evaluation. You probe the model with prompts engineered to elicit harmful, unsafe, or policy-violating output, then patch the failures you surface.
Manual red-teaming uses human testers who invent jailbreaks, role-play framings, and multi-turn attacks. It is creative and high-signal but slow and costly. You cannot scale humans to cover the attack space.
Automated red-teaming swaps the human attacker for an attacker LLM. That model generates adversarial prompts at scale. You run them against the target and track how often the target complies. The headline metric is attack success rate:
$$\text{ASR} = \frac{\text{successful attacks}}{\text{total attacks}}$$
The key limitation is the right answer here. The attacker samples from its own training distribution, so it reliably misses attack classes it never learned. A low attack success rate can mean a robust model or an under-powered attacker. You cannot tell which from the number alone.
This is why red-teaming differs from capability eval. Capability eval measures average quality on a fixed benchmark. Red-teaming hunts the worst case in an open, adversarial space. The fixes diverge too. Ensemble diverse attackers, keep a human red team in the loop, and seed attacks from real incident reports.
Quick reference
TL;DR
Red-teaming adversarially probes a model to elicit harmful outputs. Automated red-teaming uses an attacker LLM to generate attacks at scale, but inherits the attacker's own blind spots.
Think of how a bank, before it opens, hires people to try to break in: pick the locks, slip past the cameras, talk their way past the guard. Every break-in they pull off reveals a hole the bank then fixes. Red-teaming an AI is the same idea. Testers try to trick the model into saying something harmful, and each successful trick exposes a weakness to patch. Doing this by hand is slow, so people now use a second AI as the burglar that invents thousands of break-in attempts automatically. The catch: a robot burglar only knows the tricks it was taught, so it keeps missing attacks it never learned about. A human burglar with a wild imagination still finds doors the robot never thinks to try.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Relying on automated red-teaming alone for a high-stakes safety sign-off The attacker's blind spots produce false assurance. Pair it with a human red team and real incident seeds.
Measuring average product quality or helpfulness on typical traffic That is capability eval. Use a representative benchmark, not adversarial worst case search.
A single attacker model is your only attack source One distribution misses whole attack classes. Ensemble diverse attackers and report attack diversity.
The harm classifier scoring responses is unvalidated Its false negatives silently undercount unsafe outputs, inflating apparent safety regardless of attack quality.
Quick scan · 2–4 min
Everything important, quickly.
4 min: define red-teaming as worst case adversarial search, contrast manual vs automated, explain the attacker blind spot, the attack success rate metric, and how it differs from capability eval.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic and OpenAI publish red-team reports for Claude Opus 4.7 and GPT-5.5 as part of their system cards before release.
2Microsoft PyRIT automates adversarial prompt generation against target LLMs and scores harmful responses.
3HarmBench provides a standardized attack set and an attack success rate leaderboard for evaluating jailbreak robustness.
4Garak runs scripted probes (prompt injection, toxicity, leakage) as a continuous LLM vulnerability scanner.
5The GCG suffix attack demonstrated gradient-based automated jailbreaks transferring across open-weight models like Llama 4.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can a near-zero attack success rate be more worrying than a moderate one?
Sign in or go Pro to see the approach for this follow-up.
QHow would you reduce the attacker LLM's distributional blind spots?
Sign in or go Pro to see the approach for this follow-up.
QHow does red-teaming connect to the safety training loop, not just evaluation?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating a clean automated red-team pass as proof of safety. The attacker LLM only probes its own distribution, so zero successful attacks can mean blind spots, not robustness.