What is red teaming in LLM safety evaluation and how does automated red teaming scale it?

**Red-teaming** is the practice of deliberately attacking your own model to find where it breaks. In safety evaluation, that means crafting prompts designed to make the model produce harmful, toxic, or policy-violating output. Then you can find and fix the weakness before a real user does. The word comes from military and security drills, where a 'red team' plays the attacker against the defending 'blue team'. For an LLM, the attacker writes adversarial prompts: jailbreak attempts, role-play tricks, encoded requests, and multi-turn manipulation. Each prompt that succeeds reveals a hole in the model's safety training. Manual red-teaming, where humans hand-craft attacks, is thorough and creative but slow and expensive. You cannot pay people to write a million attack prompts. **Automated red-teaming** fixes the scale problem. You point a second 'attacker' LLM at your target model and have it generate thousands of adversarial prompts automatically. Then you measure how often the target gives an unsafe answer. The catch, which option A names, is that the attacker LLM only knows attacks inside its own training distribution. It systematically misses whole categories of attack, so a clean automated pass can give false confidence rather than real safety.

**Red-teaming** is adversarial safety evaluation. You probe the model with prompts engineered to elicit harmful, unsafe, or policy-violating output, then patch the failures you surface. Manual red-teaming uses human testers who invent jailbreaks, role-play framings, and multi-turn attacks. It is creative and high-signal but slow and costly. You cannot scale humans to cover the attack space. Automated red-teaming swaps the human attacker for an attacker LLM. That model generates adversarial prompts at scale. You run them against the target and track how often the target complies. The headline metric is attack success rate: $$\text{ASR} = \frac{\text{successful attacks}}{\text{total attacks}}$$ Option A is correct because it names the key limitation. The attacker samples from its own training distribution, so it reliably misses attack classes it never learned. A low attack success rate can mean a robust model or an under-powered attacker. You cannot tell which from the number alone. This is why red-teaming differs from capability eval. Capability eval measures average quality on a fixed benchmark. Red-teaming searches for the worst case in an open, adversarial space. The fixes differ too. Ensemble diverse attackers, keep a human red team in the loop, and seed attacks from real incident reports.