Design an automated red-teaming pipeline for LLM safety evaluation

**Red-teaming** an LLM means systematically trying to make it misbehave so you can fix the holes before users find them. For a production assistant you want this defensively: surface weaknesses, then patch them. The automated pipeline has three roles. An attacker component generates adversarial prompts, the target model answers them, and a judge labels each response as safe, borderline, or harmful. The attacker can be a separate LLM instructed to probe specific harm categories like dangerous advice, privacy leaks, or biased outputs. Diversity matters more than volume. A thousand near-identical prompts test one weakness a thousand times. You want varied phrasings, personas, and indirect framings so you cover the real attack surface across every harm category. You measure success with attack success rate: the fraction of attack prompts that elicit a harmful response above a severity threshold, broken out per category. A spike in one category points to a specific vulnerability to fix. Humans stay in the loop. People curate seed attacks, spot-check that the attacker is not generating junk, and validate the judge's high-severity calls, because automated systems miss novel harms.

I would frame this as a continuous measurement loop, not a one-time launch gate. **Three components.** An attacker generates diverse adversarial prompts per harm category. The target is the production model. A judge classifier labels each response as safe, borderline, or harmful against a per-category rubric. Diversity is the hard part. I vary personas, formats, languages, and indirect framings so coverage spans the real attack taxonomy. The headline metric is attack success rate per harm category, severity-weighted. ASR is: $$\text{ASR}_c = \frac{\#\{\text{attacks in } c \text{ breaching threshold}\}}{\#\{\text{attacks in } c\}}$$ Report each category separately. Aggregate ASR hides a dangerous category behind safe ones. I also gate releases on a frozen suite of known jailbreaks. **Human oversight is non-negotiable.** People curate seed attacks, audit attacker outputs for degeneracy, and validate judge positives and negatives. The attacker shares the target's blind spots, so its own safety training suppresses whole attack classes. Humans inject the categories the attacker will never invent.