Describe garak's role in the red-team toolchain

**Garak is an open-source LLM vulnerability scanner, think 'nmap for LLMs.'** You point it at a model endpoint and it runs a battery of probes, producing a structured findings file that lists which probes succeeded. The probe categories cover the common attack surface: - **Prompt injection**, known templates (DAN, AIM, role-override) and variants. - **Jailbreaks**, encoding attacks (base64, ROT13), prefix injection, refusal suppression. - **Data leakage**, probes designed to elicit training-data extraction or system-prompt disclosure. - **Malware and harmful content**, probes for generating malicious code, instructions for harm. - **Hallucination**, probes that test made-up entity recognition and false-fact generation. - **Toxicity and bias**, adversarial templates that elicit slurs, stereotypes, or biased reasoning. Garak's role in the red-team toolchain is first-pass discovery. Before a security team commits to a targeted red-team campaign, they run garak to catch the obvious weaknesses. The findings file becomes the input to deeper investigation: probes that succeeded indicate categories where the model is vulnerable; those categories get focused attention. Garak is open-source from NVIDIA and the broader community. It plugs into common model endpoints (Hugging Face, OpenAI-compatible APIs, REST). It runs in CI for regression detection, in periodic schedules for ongoing scanning, and on-demand during incident investigation. **What garak is not.** It is not a substitute for bespoke red-teaming. It is not continuous evaluation against your custom policies. Its probe library is public, so a model trained to pass garak's tests may still be vulnerable to novel attacks. Treat it as the floor of red-team coverage, not the ceiling.

**Garak is the open-source LLM vulnerability scanner, the nmap for LLMs that produces a first-pass findings report for any model endpoint you point it at.** **What it does.** Garak runs a battery of probes against a model endpoint and reports which probes succeeded. The probe library covers prompt injection (direct templates, indirect via tool-output simulation), jailbreaks (encoding attacks, prefix injection, refusal suppression, persona overrides), data leakage (training-data extraction, system-prompt extraction), harmful generation (malware, weapons, illicit instructions), hallucination (fabricated-entity recognition), and toxicity and bias adversarial probing. Each probe has a generator (the prompt to send) and a detector (how to decide whether the response indicates success). The output is a structured JSON findings file: per-probe success rates, example successful responses, severity classification. The findings file is consumable by humans (red-team triage) and by tooling (regression detection, CI gating). **Where it fits in the red-team toolchain.** Three roles. First, *first-pass discovery.* Before a security team commits to a targeted red-team campaign, they run garak. The findings highlight which attack categories the model is vulnerable to; the campaign then focuses on those categories rather than starting from scratch. Second, *CI regression.* After model upgrades, fine-tune refreshes, or prompt changes, run garak in CI to catch regressions. A previously-passing probe that now succeeds indicates a regression. Third, *periodic ongoing scanning.* On a schedule (weekly, monthly), run garak against production endpoints to catch drift or new probe-template additions in the garak library. **Comparison with PyRIT.** Garak is a scanner with a fixed probe library; PyRIT (Microsoft's Python Risk Identification Tool) is a framework for building custom red-team campaigns.