Best fit: simulated adaptive attacker over a multi-turn agent
PyRIT and garak are the two most prominent open-source red-team tools for LLMs, and they are aimed at different jobs.
**garak** is from NVIDIA. It is a vulnerability scanner. It ships with a library of pre-built probes, encoded-prompt attacks, malware generation attempts, training-data extraction probes, hallucination probes, and many more. You point it at a model endpoint and run it as a CLI sweep, and it produces a structured pass-fail report per probe. It is the right tool for a CI regression suite where you want to verify on every release that known attack patterns still fail.
**PyRIT** is from Microsoft. It is an orchestration framework. It does not have a fixed probe library; instead it provides building blocks for adaptive attacks, attacker LLMs that try, observe, and adjust; multi-turn conversation state; scorers that evaluate target responses; target adapters across Azure, OpenAI, Anthropic, and other endpoints. You write a red-team scenario in PyRIT and it runs an adaptive attack against your system. It is the right tool for a quarterly adversarial exercise where you want to simulate a real attacker who thinks.
The production answer is to use both. garak in CI catches regressions on known attacks. PyRIT in scheduled exercises finds the novel adaptive shapes garak's static library would never catch. The matches in this question are exactly that mapping.
**PyRIT and garak occupy complementary roles in a production red-team programme.**
**garak (NVIDIA)** is a **vulnerability scanner**. Its value is in its **library of pre-built probes** organised by attack class, encoding (base64, leetspeak, ROT13), DAN-family persona jailbreaks, refusal-suppression, prefix injection, training-data extraction, malware-generation, hallucination, latent-injection, glitch-token, and many more. Each probe is deterministic, has a pass-fail outcome, and can be run from the command line against any HTTP-accessible target. It is the right tool for **regression testing in CI**, verify every release that no known attack class regressed, produce a structured report, gate the release on attack success rate per class.
**PyRIT (Microsoft)** is an **orchestration framework**. Its value is in the **machinery for adaptive attacks**, an attacker LLM that proposes, observes, and adapts; conversation-state machinery for multi-turn attacks; scorers (often LLM-as-judge) that evaluate target responses against goals; target adapters across the major model and platform endpoints. It is the right tool for **simulated adaptive attacker exercises**, quarterly red-team sprints where you want to know what a thinking adversary could achieve against your agent, including stacked attacks and multi-turn social engineering that a static probe library cannot encode.
**Why the production answer is both.**
- garak catches regressions on known patterns at high volume and low cost.
- PyRIT discovers novel attack shapes that should then be encoded as garak probes for future regression.
- The two form a feedback loop: PyRIT finds something, becomes a garak probe, gets run in CI on every release.
**Common confusions to avoid.** garak's adaptive_probes plugin offers some dynamic behaviour, and PyRIT can be scripted into a CI-style sweep, so the boundary is not absolute.
Quick reference
TL;DR
PyRIT (Microsoft) is an orchestration framework for adaptive multi-turn attacker-LLM red-teaming; garak (NVIDIA) is a scanner with a fixed probe library, use garak in CI regression, use PyRIT for quarterly
Think of two tools in a security team's kit. The first is like an antivirus scanner, it has a big library of known threats and you run it on a schedule to check that none of them slip through. That is garak. The second is like a hired penetration tester who actually thinks during the test, tries one trick, watches your reaction, tries a different angle, and chains attacks together. That is PyRIT. You need both. The scanner catches the regressions you would never notice; the tester finds the surprises that no static list could have predicted.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very small applications with low risk and no agent surface Even garak in CI may be overkill; a small curated regression set is enough.
Strictly batch-mode offline pipelines with no adversarial input Both tools assume an adversary; benign pipelines do not need adversarial scanning.
Models served by a provider that already publishes detailed adversarial evaluations Some provider evaluation results substitute for breadth coverage; supplement with depth-only exercises.
Quick scan · 2–4 min
Everything important, quickly.
3 min: contrast a probe-library scanner with an orchestration framework, describe the CI versus scenario fit for each, and walk through the feedback loop from PyRIT findings to garak probes.
Real-world examples
Real products, models, and research that use this idea.
1Microsoft red-team teams publish PyRIT scenarios that simulate adaptive attacker behaviour against Azure OpenAI deployments; these have driven several public CVE-like advisories on prompt-injection patterns.
2NVIDIA's garak is bundled into NeMo Guardrails CI examples and is increasingly the default open-source regression scanner for OSS LLM application stacks.
3Commercial red-team services like Lakera Red and Patronus typically combine elements of both, pre-built probe suites plus adaptive attacker harnesses, recognising that neither layer alone is sufficient.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you turn a PyRIT-discovered incident into a garak probe?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right cadence for PyRIT exercises versus garak regression?
Sign in or go Pro to see the approach for this follow-up.
QHow do you red-team a multi-agent system?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating PyRIT and garak as alternatives to each other; they cover different parts of the red-team programme and the production answer is to use both with clear roles.