AI Agent Security — the threat model nobody shipped with
Autonomous agents are a new attack surface: they read untrusted content, keep long-term memory, and call tools that mutate the real world. Classic application security doesn't cover this. This page is a practical threat model for anyone running an autonomous agent — and how to close the gaps.
Every agent architecture makes the same mistake: it lets retrieved content grant permission to act. Treat retrieval and permission as separate, independently-verified decisions and most agent attacks collapse.
The new attack surface
A traditional app has a fixed set of inputs and a server-side enforcement layer. An autonomous agent is different — it takes unstructured, untrusted input (documents, emails, web pages, tool outputs) and converts it into actions (API calls, file writes, tool invocations) without a human in the loop. Every one of those inputs is a potential attacker, and the boundary between "read" and "act" is porous.
Threat model: what can go wrong
Instruction override
A malicious document tells the agent to ignore its system prompt and exfiltrate data or run a destructive tool.
Persistent false facts
False "facts" are written to long-term memory and corrupt every future decision for weeks.
Secret leakage
API keys and tokens persist into memory, then get sent out by the next injection.
Malicious tool call
The agent is steered into calling delete_file, transfer_funds, or post_public with attacker-controlled arguments.
Fake privilege
Content that claims to come from a trusted source gains unwarranted power over the agent's actions.
MCP & plugin risk
A compromised tool server or MCP endpoint becomes a vector into every agent that trusts it.
The three boundaries you must secure
- Ingest boundary — what content gets into context. Hard to secure (the model must read to be useful).
- Memory boundary — what content gets persisted as trusted memory. Securing this stops poisoning at the source.
- Action boundary — what content gets to cause an action. Enforcing here is the last line of defense and the one that actually matters.
A practical baseline for any agent
- Bind authority to origin — content from untrusted sources must not carry trusted-source privileges.
- Enforce retrieval ≠ permission — reading context never, by itself, authorises acting on it.
- Revalidate at action time — re-score the actual action against its provenance before it executes.
- Redact credentials before persistence — secrets never live in memory in usable form.
- Fail closed — when unsure or tampered, block rather than allow.
- Separate privileges — an operator key and an agent runtime key must not be interchangeable.