What is Jailbreak?

How It Works

A Jailbreak is defined by its goal and the policy being tested, not by one prompt pattern. The target may be a base model, hosted assistant, multimodal model, or Agent that combines a model with tools. The relevant boundary is an explicit Safety Policy: a successful test must cause behavior that the evaluated policy says should be refused or safely redirected. Unusual wording, toxicity, non-refusal, or a model discussing a sensitive topic is not enough by itself. The same output may be allowed in one policy context and disallowed in another.

Jailbreak and Prompt Injection overlap, and taxonomies do not draw the boundary identically. OWASP LLM01:2025 treats jailbreaking as a form of Prompt Injection whose goal is to make a model disregard safety protocols. For engineering analysis, record three dimensions separately: Delivery Vector, such as direct user input, external content, an image, or Fine-tuning; Mechanism, such as role or context reframing, multi-turn escalation, many-shot demonstrations, encoding, token optimization, multimodal perturbation, or automated search; and Impact, such as disallowed text, data disclosure, a Tool Call, persistent state change, or an External Effect. Not every Prompt Injection is a Jailbreak, and a Jailbreak payload can arrive through an indirect injection.

A Model Policy Bypass does not automatically create application privilege. Real impact depends on which data, tools, credentials, network destinations, and approvals the surrounding system exposes. Conversely, an Agent can turn a seemingly textual safety failure into a consequential side effect if application code accepts model proposals without deterministic Authorization. Model-level safety and application security therefore need separate controls and separate acceptance criteria.

Reproducible evaluation must bind the exact Model ID and Revision, System Prompt Hash, Chat Template, Safety Policy Revision, Guardrail Revision, Behavior Dataset and Split, attack artifacts, Delivery Vector, attacker access, query and turn budget, Sampling Configuration, Seed when applicable, Tool Mode, Judge ID and Revision, and Human Review protocol. Attack Success Rate is not comparable when these fields differ. Keep test cases isolated from training and tuning data, preserve failed attempts and costs, and evaluate single-turn, multi-turn, multilingual, obfuscated, multimodal, and adaptive cases only where the Threat Model supports them.

Do not define success as absence of refusal text. Measure whether the response gives specific and useful disallowed assistance, partially leaks restricted information, proposes or executes a prohibited action, or causes an External Effect. Report these dimensions separately from Benign Refusal Rate, safe-redirection quality, normal-task utility, latency, token cost, and classifier false positives. Automated judges can be biased, vulnerable to adversarial content, or changed by a provider; calibrate them against blinded Human Review and store the exact Judge Revision.

Defense in Depth combines model safety training and adversarial training with input and exchange classification, Output Moderation, rate limits, anomaly detection, and continuous regression testing. Application controls must still validate tool arguments in deterministic code, enforce Least Privilege, isolate secrets, constrain network egress, sandbox execution, require Human Approval for consequential effects, and stop unsafe output before delivery or action. Every layer has bypass and False Refusal trade-offs, so no filter, system prompt, classifier, benchmark score, or model family proves complete prevention.

Jailbreak testing is dual-use security work. Run it only against systems you own or are explicitly authorized to assess, in an isolated environment with tools and outbound network disabled by default. Store harmful corpora and outputs under access control, minimize exposure in logs and reports, define stop conditions, route severe findings through Responsible Disclosure, and connect regression failures to release blocking and Incident Response. Public documentation should use abstract attack families and hashes rather than reusable harmful payloads.

Key Characteristics

• Policy-relative outcome — success means violating the declared Safety Policy, not merely producing unusual text or omitting a refusal phrase
• Multiple delivery surfaces — direct input, untrusted content, multimodal data, conversation history, and model customization can carry or create the bypass
• Adaptive attack budget — single-turn templates, multi-turn search, automated optimization, and repeated queries have different capabilities and costs
• System-dependent impact — harmful text, data access, Tool Calls, and external side effects depend on the surrounding Agent permissions and controls
• Evaluator-sensitive results — ASR changes with the behavior set, policy, decoding, Judge, success rubric, model revision, and test contamination
• Defense-utility trade-off — stronger filtering may reduce attacks while increasing False Refusal, latency, cost, or loss of benign capability

Common Use Cases

1. Authorized Red Teaming — test a declared model and application policy before release using isolated, access-controlled artifacts
2. Safety regression gates — rerun versioned behavior suites when the model, system prompt, guardrail, tool set, or policy changes
3. Defense evaluation — compare model training, classifiers, moderation, and application controls under the same adaptive Threat Model
4. Agent impact testing — verify that a model safety failure cannot bypass deterministic Authorization or produce unauthorized side effects
5. Incident analysis — preserve prompt and response hashes, model identity, policy revision, tool traces, and Judge evidence for triage and disclosure

Frequently Asked Questions

Related Terms

Related Articles