Match PyRIT and garak to the red teaming jobs they are actually good at

PyRIT and garak are the two most prominent open-source red-team tools for LLMs, and they are aimed at different jobs. **garak** is from NVIDIA. It is a vulnerability scanner. It ships with a library of pre-built probes, encoded-prompt attacks, malware generation attempts, training-data extraction probes, hallucination probes, and many more. You point it at a model endpoint and run it as a CLI sweep, and it produces a structured pass-fail report per probe. It is the right tool for a CI regression suite where you want to verify on every release that known attack patterns still fail. **PyRIT** is from Microsoft. It is an orchestration framework. It does not have a fixed probe library; instead it provides building blocks for adaptive attacks, attacker LLMs that try, observe, and adjust; multi-turn conversation state; scorers that evaluate target responses; target adapters across Azure, OpenAI, Anthropic, and other endpoints. You write a red-team scenario in PyRIT and it runs an adaptive attack against your system. It is the right tool for a quarterly adversarial exercise where you want to simulate a real attacker who thinks. The production answer is to use both. garak in CI catches regressions on known attacks. PyRIT in scheduled exercises finds the novel adaptive shapes garak's static library would never catch. The matches in this question are exactly that mapping.

**PyRIT and garak occupy complementary roles in a production red-team programme.** **garak (NVIDIA)** is a **vulnerability scanner**. Its value is in its **library of pre-built probes** organised by attack class, encoding (base64, leetspeak, ROT13), DAN-family persona jailbreaks, refusal-suppression, prefix injection, training-data extraction, malware-generation, hallucination, latent-injection, glitch-token, and many more. Each probe is deterministic, has a pass-fail outcome, and can be run from the command line against any HTTP-accessible target. It is the right tool for **regression testing in CI**, verify every release that no known attack class regressed, produce a structured report, gate the release on attack success rate per class. **PyRIT (Microsoft)** is an **orchestration framework**. Its value is in the **machinery for adaptive attacks**, an attacker LLM that proposes, observes, and adapts; conversation-state machinery for multi-turn attacks; scorers (often LLM-as-judge) that evaluate target responses against goals; target adapters across the major model and platform endpoints. It is the right tool for **simulated adaptive attacker exercises**, quarterly red-team sprints where you want to know what a thinking adversary could achieve against your agent, including stacked attacks and multi-turn social engineering that a static probe library cannot encode. **Why the production answer is both.** - garak catches regressions on known patterns at high volume and low cost. - PyRIT discovers novel attack shapes that should then be encoded as garak probes for future regression. - The two form a feedback loop: PyRIT finds something, becomes a garak probe, gets run in CI on every release. **Common confusions to avoid.** garak's adaptive_probes plugin offers some dynamic behaviour, and PyRIT can be scripted into a CI-style sweep, so the boundary is not absolute.