What is Prompt Injection?
Prompt injection is attacker-controlled content that causes an LLM application to follow an unintended instruction or pursue an unintended goal. It becomes a security incident when that influence reaches private data, tools, persistent state, or an external destination.
Quick Facts
| Full Name | Prompt Injection Attack |
|---|---|
| Created | Widely discovered with the opening of LLM APIs like ChatGPT in 2022, and listed as the #1 vulnerability in the OWASP LLM Top 10 in 2023. |
| Specification | Official Specification |
How It Works
Prompt injection is a confused-deputy problem. A model is allowed to interpret content or act for a user, then attacker-controlled text influences it to use that authority for another purpose. Direct injection comes from the user request. Indirect injection can arrive through web pages, emails, RAG chunks, source code, tool results, images, OCR, metadata, or prior memory. Labels, role boundaries, and classifiers improve behavior but are not deterministic security boundaries. Design for the possibility that untrusted content influences the model: minimize capabilities, preserve source provenance, enforce authentication and object-level authorization in code, control data egress, bind confirmations to exact actions, and test complete attack chains. See the Prompt Injection Defense guide for the production architecture.
Key Characteristics
- Direct and indirect vectors: hostile content can be typed, retrieved, rendered, or persisted
- Impact depends on capabilities: the same text is lower risk in a read-only summarizer than in an agent with private data and external tools
- Role separation helps model behavior but does not replace authorization
- Keyword, regex, and classifier controls are useful signals but can be bypassed or create false positives
- Security controls must cover data flow, tool arguments, destinations, and durable state, not only response text
Common Use Cases
- Threat modeling agents that read untrusted sources and can invoke tools
- Testing RAG, browser, email, coding, and MCP workflows with indirect injection cases
- Auditing tool permissions, resource ownership, external destinations, and approval paths
- Building provenance-aware policy gates for sensitive tool arguments
- Turning confirmed incidents into regression tests and incident-response playbooks
Example
Loading code...Frequently Asked Questions
What is the difference between Prompt Injection and a Jailbreak?
While often used interchangeably, the focus differs. **Jailbreaking** usually refers to attacking the foundational model itself (like chatting directly with ChatGPT) to bypass the ethical and safety guardrails set by the creator (e.g., OpenAI/Anthropic) to make it write malware. **Prompt Injection** generally targets a developer-built 'LLM Application', aiming to overwrite the developer's custom System Prompt and hijack the application's business logic.
What is Indirect Prompt Injection?
This is a more stealthy attack vector. The attacker does not chat directly with your AI bot. Instead, they write malicious instructions on their public webpage, in a resume PDF, or in an email (even hiding it with white text). When a victim's AI assistant (like Microsoft Copilot or a RAG-based resume screener) reads this external content, the AI treats the malicious instructions as legitimate commands, potentially secretly forwarding summaries to the attacker's email.
How can I effectively defend against Prompt Injection?
There is no single perfect defense; you must use defense-in-depth: 1. **Instruction Segregation**: Use model-native specific delimiters or role tags (like OpenAI's ChatML format) to clearly separate system and user contexts. 2. **Principle of Least Privilege**: Do not grant AI Agents high-risk tools (like direct SQL execution or email deletion); all high-risk operations must have a Human-in-the-loop for confirmation. 3. **LLM Firewalls**: Add a specialized, security-trained small model (like Llama Guard) at the input and output stages to detect malicious patterns.