Prompt Injection Defense: Secure LLM Agents by Design

Prompt injection is not primarily a bad-string problem. It is a confused-deputy problem: a model is authorized to process a user’s data or act on a user’s behalf, then attacker-controlled content persuades it to use that authority for a different purpose.

The same injected sentence has radically different impact in two systems. A read-only summarizer may produce a bad summary. An agent that can read private mail, upload files, write memory, call MCP tools, and access the public internet may leak data or create persistent compromise.

The engineering goal is therefore not “make the model impossible to manipulate.” It is:

Even if untrusted content influences the model, the system must prevent unauthorized data access, state changes, and external effects.

This guide builds that architecture from the threat model outward. Claims and source guidance were reviewed on July 16, 2026.

Key Takeaways

• Treat every model output as an untrusted proposal, including tool calls and generated code.
• Separate trusted instructions from untrusted data, but do not mistake delimiters for a security boundary.
• Keep authentication, authorization, validation, rate limits, and policy enforcement in deterministic code.
• Break dangerous capability combinations: untrusted input, sensitive data, and an external communication channel should not coexist without controls.
• Track provenance through derived values; a summary of an untrusted page is still influenced by that page.
• Bind user approval to the exact action and arguments shown, not to a vague “continue?” prompt.
• Constrain network egress, redirects, URLs, rendered Markdown, and other covert output channels.
• Test utility and security together against adaptive, repeated, multi-channel attacks.

What Prompt Injection Is

OWASP LLM01:2025 defines prompt injection as input that changes model behavior or output in unintended ways. The input can be visible or hidden, direct or delivered through external content. NIST AI 100-2 E2025 places prompt injection within a broader adversarial machine-learning taxonomy.

The useful distinction is not simply “system text versus user text.” Modern APIs do provide role and instruction hierarchies, and models can be trained to follow them more reliably. The security problem remains probabilistic: lower-trust text can still influence decisions, especially when the task itself requires interpreting that text.

Prompt Injection Versus Jailbreaking

• Prompt injection: redirects application behavior. Example: a malicious email tells an assistant to forward private attachments.
• Jailbreaking: bypasses model safety policy. Example: a user tries to elicit prohibited content.

They overlap, but their controls differ. Model safety training matters heavily for jailbreaks. Agent prompt injection also demands conventional application security because the impact depends on data and tools outside the model.

Direct and Indirect Injection

Direct injection arrives through the user-controlled request. It can request instruction override, secret extraction, unauthorized tool use, or policy evasion.

Indirect injection arrives through data the system reads:

• web pages, search results, advertisements, and link metadata;
• email, calendars, chat messages, tickets, and documents;
• source repositories, issue comments, build logs, and package metadata;
• RAG chunks and poisoned knowledge bases;
• tool, MCP server, sub-agent, and API responses;
• images, audio, OCR layers, invisible DOM, and document metadata;
• previously written memory or summaries.

Indirect injection is difficult because the hostile content may also contain information the user legitimately needs. Removing all instructions can destroy the task.

The Attack Chain

A meaningful security incident usually requires more than a successful instruction override:

attacker-controlled source
        |
        v
model interprets content as authority
        |
        v
agent accesses a capability or private asset
        |
        v
unauthorized action, disclosure, or persistent state

Defenders can break any edge:

1. prevent or label attacker-controlled content;
2. reduce its authority in planning;
3. deny unnecessary capabilities;
4. authorize every operation against the real user and resource;
5. block unsafe data flows and destinations;
6. require meaningful human confirmation;
7. prevent untrusted results from becoming trusted memory.

Model robustness is one layer, not the whole control system.

Attack Classes a Production Test Must Cover

Instruction and Goal Hijacking

The model abandons the user’s task, changes selection criteria, misrepresents a document, or follows a new objective. No tool call is required for this to harm ranking, moderation, hiring, or recommendation systems.

Data Exfiltration

Injected content induces the agent to retrieve secrets and transmit them through:

• tool arguments or message recipients;
• attacker-controlled URLs and query strings;
• image, link-preview, Markdown, or redirect requests;
• generated files, forms, comments, or public posts;
• side effects visible to another tenant.

Output text filtering alone misses these non-text channels.

Tool and Agent Hijacking

The attack invokes an allowed tool for an unauthorized purpose, changes arguments, causes repeated actions, or delegates to a sub-agent with broader authority. A valid JSON tool call can still be malicious.

RAG and Corpus Poisoning

An attacker inserts content likely to rank for target queries. The payload may manipulate the answer, request secrets, or influence future memory. Retrieval relevance and content trust are separate dimensions.

Persistent and Delayed Injection

The payload writes itself into memory, project instructions, summaries, workflow state, or generated configuration. It activates in a later session, after the original source is no longer visible.

Multimodal and Obfuscated Injection

Instructions can appear in images, OCR text, audio, Unicode, encoded blocks, whitespace, CSS, or metadata. Normalization can expose some payloads but cannot determine intent reliably.

Best-of-N and Adaptive Attacks

An attacker varies wording, language, encoding, placement, and multi-turn setup until one attempt works. A defense that passes one static payload is not a security result.

Threat Model Before Controls

Document these elements per workflow:

Example invariant:

Content from an email may influence its summary, but cannot select a new recipient for private attachments.

This is enforceable in code. “The model must ignore malicious instructions” is not.

Why Common Defenses Fail

Keyword and Regex Blocking

Patterns such as ignore previous instructions catch demonstrations, not the attack class. Attackers can paraphrase or distribute the objective across context. Legitimate security discussions also contain those phrases, causing false positives.

Use detectors for telemetry, triage, and one defense layer. Never grant authority because a detector returned “safe.”

Destructive Sanitization

Deleting braces, angle brackets, Unicode, or Base64 breaks valid code and documents while leaving ordinary-language attacks intact. Sanitization is valuable for removing active HTML, scripts, macros, or unsafe file constructs; it cannot reliably sanitize meaning.

Delimiters and Prompt Hardening

Real system/developer/user/tool roles and clear labels help model behavior. Handwritten strings such as <|system|> inside one prompt do not create provider-level roles. Even genuine role separation remains a probabilistic model control, not deterministic authorization.

Keeping the System Prompt Secret

Do not place credentials or security-critical secrets in prompts. Assume instructions may be reconstructed or leaked. Prompt confidentiality can protect intellectual property, but it must not be required for authorization.

Output Word Filters

Searching for PASSWORD or the first 50 characters of a prompt misses paraphrases and tool-mediated exfiltration, while blocking benign explanations. Enforce schemas and inspect data flows, destinations, and rendered output instead.

A “Guardrail Model” Alone

Classifiers and hardened models raise attack cost, but they share probabilistic and distribution-shift limitations. Anthropic’s browser-agent research explicitly reports residual risk after substantial improvements. Security-sensitive controls must remain outside the classifier.

Secure Architecture

1. Minimize Agency

If a workflow only summarizes, provide no write tools. If it needs one calendar, do not grant all drives and mailboxes. Use short-lived, task-scoped credentials and preserve source-system authorization.

Prefer:

summarize selected emails

over:

read all mail + read all files + browse anywhere + send messages

The safest dangerous capability is the one the process never receives.

2. Separate Control Flow From Untrusted Data

Build the plan from trusted user intent and policy. Process untrusted content in a quarantined component without privileged tools. Return typed values, not free-form instructions, to the privileged controller.

This pattern still needs data-flow policy: an attacker may not control which tool is called but could manipulate an argument such as the destination address. CaMeL research extends the dual-model idea with provenance and capabilities so policies can constrain both control and data flows.

3. Track Provenance and Trust

Attach origin to data:

recipient = "attacker.example"
origin = web_page_73
trust = untrusted
derived_by = extraction_model_v4

Taint should propagate through summaries, translations, OCR, and model extraction. Transformation does not make attacker-controlled information trusted.

4. Authorize Tool Calls Deterministically

Tool name validation is not enough. Check:

• authenticated principal and tenant;
• resource ownership and object-level permission;
• allowed action and parameter schema;
• data classification and destination;
• provenance of each security-sensitive argument;
• transaction, rate, and monetary limits;
• whether exact user confirmation is required.

The model proposes. Application code decides.

5. Control Egress

Use destination allowlists, DNS/IP validation, redirect limits, URL revalidation after redirects, and network policy. Prevent sensitive values from entering URLs, rendered Markdown, analytics, logs, or public tool arguments. Proxy external fetches rather than giving model-generated code unrestricted networking.

6. Bind Confirmation to the Action

“The agent needs permission to continue” is not informed consent. Show the exact recipient, resource, amount, data fields, and irreversible effect. Bind approval cryptographically or transactionally to the canonical action so the model cannot change arguments afterward.

7. Sandbox Execution

Generated code needs an isolated runtime with:

• no ambient credentials;
• read-only or ephemeral filesystem by default;
• explicit CPU, memory, time, process, and output limits;
• disabled network or narrow egress proxy;
• audited package and syscall policy;
• no direct path from untrusted content to host execution.

Sandboxing limits impact; it does not decide whether the intended action was authorized.

8. Protect RAG and Memory

• enforce document ACLs before retrieval;
• retain source identity, author, time, and trust;
• separate relevance score from trust score;
• never promote retrieved instructions into procedural memory automatically;
• quarantine new corpus content and re-evaluate when parsers change;
• make persistent memory writes a separately authorized operation;
• propagate deletion to indexes, summaries, caches, and memory.

Runnable Deterministic Policy Gate

This standard-library example guards action execution after a model proposes a call. It is intentionally independent of any LLM SDK.

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum
from hashlib import sha256
import json


class Verdict(str, Enum):
    ALLOW = "allow"
    CONFIRM = "confirm"
    DENY = "deny"


@dataclass(frozen=True)
class ToolCall:
    user_id: str
    action: str
    resource_owner: str
    destination: str | None
    data_labels: frozenset[str]
    influenced_by_untrusted: bool
    destructive: bool = False


@dataclass(frozen=True)
class Decision:
    verdict: Verdict
    reason: str
    action_digest: str


def action_digest(call: ToolCall) -> str:
    payload = {
        "action": call.action,
        "data_labels": sorted(call.data_labels),
        "destination": call.destination,
        "destructive": call.destructive,
        "resource_owner": call.resource_owner,
        "user_id": call.user_id,
    }
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return sha256(canonical.encode()).hexdigest()


def authorize(
    call: ToolCall,
    *,
    allowed_actions: set[str],
    trusted_destinations: set[str],
    approved_digest: str | None = None,
) -> Decision:
    digest = action_digest(call)

    if call.action not in allowed_actions:
        return Decision(Verdict.DENY, "action is outside task scope", digest)
    if call.resource_owner != call.user_id:
        return Decision(Verdict.DENY, "resource owner mismatch", digest)

    external = (
        call.destination is not None
        and call.destination not in trusted_destinations
    )
    sensitive = bool(call.data_labels & {"secret", "personal", "financial"})

    if call.influenced_by_untrusted and (external or call.destructive):
        return Decision(
            Verdict.DENY,
            "untrusted content cannot control external or destructive effects",
            digest,
        )
    if external and sensitive:
        return Decision(
            Verdict.DENY,
            "sensitive data cannot be sent to an untrusted destination",
            digest,
        )
    if external or call.destructive:
        if approved_digest != digest:
            return Decision(
                Verdict.CONFIRM,
                "exact action requires user confirmation",
                digest,
            )

    return Decision(Verdict.ALLOW, "policy satisfied", digest)

The policy is illustrative, not universal. A production implementation needs centrally managed identity, tenant-scoped object authorization, canonical destination parsing (including redirect and DNS policy), durable rate limits, transactional approvals, and audit logs. A string set is not proof that a destination is safe. Its important property is that changing the recipient or resource changes the digest and invalidates prior approval.

Security Evaluation

Build Paired Utility and Attack Sets

For every workflow, include:

• normal tasks and edge cases;
• direct and indirect attacks;
• each untrusted source and modality;
• tool-argument manipulation;
• private-data exfiltration;
• persistent memory and delayed activation;
• multilingual, encoded, and obfuscated variants;
• multi-turn and Best-of-N attempts;
• attacks combined with malformed tool output and redirects.

Measure Outcomes, Not Refusal Words

Track:

• Attack Success Rate: unauthorized objective achieved;
• Utility Success Rate: legitimate task completed correctly;
• unauthorized tool-call and data-egress rate;
• cross-tenant access rate;
• confirmation precision and user cancellation;
• detector false-positive and false-negative rates;
• p50/p95 latency and cost;
• repeated-attempt success with confidence intervals.

A model saying “I cannot do that” is irrelevant if it already sent a request. Conversely, mentioning an injection in a safe summary is not an attack success.

Test Adaptively

Static payload suites become stale. Allow an authorized red-team agent or human tester to observe defenses and mutate attacks within a strict sandbox. Rerun the suite for every model, prompt, parser, tool, MCP server, memory, retrieval, and policy change.

Keep production canaries and incident telemetry, but redact sensitive prompt content and protect logs from becoming a new data leak.

Incident Response

When an injection causes or may have caused impact:

1. stop affected tools, connectors, and outbound routes;
2. revoke task credentials and session tokens;
3. preserve prompts, source content, tool calls, approvals, and policy decisions;
4. identify affected users, tenants, resources, and destinations;
5. remove poisoned documents, memory, summaries, caches, and indexes;
6. verify downstream actions and roll back where possible;
7. patch the architectural path, then add the chain to regression tests;
8. notify users or regulators according to incident policy.

Deleting the visible payload is insufficient if a derived summary or memory still contains its influence.

Production Checklist

• [ ] Assets, principals, sources, capabilities, sinks, and invariants are documented.
• [ ] Every external source and tool result is marked untrusted by default.
• [ ] The model has no ambient authority.
• [ ] Tool authorization uses the real user, tenant, resource, and action.
• [ ] Security-sensitive arguments carry provenance.
• [ ] Sensitive data cannot reach arbitrary destinations.
• [ ] Confirmations show and bind the exact effect.
• [ ] Generated code runs in a constrained sandbox.
• [ ] RAG ACLs are enforced before semantic ranking.
• [ ] Memory and configuration writes are separately authorized.
• [ ] Utility and adaptive attack suites gate every release.
• [ ] Incident response covers derived state and credential revocation.

Frequently Asked Questions

Is a system prompt an access-control boundary?

No. Use it to state behavior, not to hold credentials or enforce permissions. Backend systems must reject unauthorized operations regardless of the generated text.

Should suspicious input always be blocked?

No. A security analyst may legitimately ask the model to analyze an injection payload. Label the source, restrict capabilities, and evaluate intended effects rather than blocking phrases blindly.

Does fine-tuning or RAG solve prompt injection?

No. They may improve behavior for a distribution but do not create deterministic separation between instructions and untrusted data. RAG also adds an indirect-input surface.

Are commercial models automatically safer than open models?

Security depends on the exact model, configuration, system architecture, tools, credentials, and threat model. Provider defenses help, but application owners still control the consequential capabilities.

Conclusion

Prompt injection should be treated as an expected compromise of a probabilistic decision component. Secure systems do not ask that component to authorize itself.

Reduce available power, preserve provenance, enforce object-level policy in code, constrain data egress, bind approvals, and continuously test complete attack chains. The model can remain useful even when the surrounding system refuses to let untrusted language become authority.

Primary Sources

• OWASP LLM Prompt Injection Prevention Cheat Sheet
• OWASP LLM01:2025 Prompt Injection
• NIST AI 100-2 E2025: Adversarial Machine Learning Taxonomy
• CaMeL: Defeating Prompt Injections by Design
• Anthropic: Mitigating Prompt Injections in Browser Use
• OpenAI: Designing AI Agents to Resist Prompt Injection