Jailbreak
Also known as: Jailbreaking, Safety bypass
Adversarial prompt that bypasses an LLM's safety training to elicit normally-refused content.
An adversarial prompting technique that bypasses an LLM's safety training to elicit normally-refused content. Methods range from roleplay framings (DAN) to encoded payloads, multi-turn manipulation, and gradient-based attacks (GCG).
In practice
Constant cat-and-mouse for any consumer-facing app. Senior safety interviews probe defense-in-depth and the limits of RLHF safety.
How it compares
Jailbreak targets the model's safety policy; prompt injection targets the system prompt and tool integrity.
Related topics
Questions that mention this term
- A consumer facing LLM product is about to launch. Describe the safety evaluation that should run first.
- Walk through a multimodal jailbreak via image embedded instructions
- Design an automated red teaming pipeline for LLM safety evaluation
- Identify the refusal suppression pattern in this user prompt
- Pick the right classifier…
- Tradeoffs of exposing a reasoning model's full chain-of-thought to end users.
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.