Red Teaming
Also known as: Adversarial testing
Adversarially probe the model/app to find safety failures, jailbreaks, and prompt-injection holes.
Systematic adversarial probing of a model or LLM app to surface harmful outputs, bypass safety filters, or exploit prompt injection. Combines manual creativity with automated attack generation.
In practice
Standard pre-launch step for any user-facing LLM app. Safety interviews probe attack categories and coverage measurement.
How it compares
Prompt injection is one attack class; red teaming is the broader practice of finding any safety failures.
Related topics
Questions that mention this term
- A consumer facing LLM product is about to launch. Describe the safety evaluation that should run first.
- Pick where a continuous…
- Design an automated red teaming pipeline for LLM safety evaluation
- What category do Patronus AI and Vijil occupy in the 2026 stack
- What is red teaming in LLM safety evaluation and how does automated red teaming scale it?
- Describe garak's role in the red team toolchain
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.