The Futility of Threat Modeling AI Agents

There’s been a lot said recently about how to secure AI agents with every technology from sandboxes to identity touted as the solution to all of your problems. But what problems to each of these technologies address? Which of them do you actually need? And how do they fit together into a single system that makes your production AI agents secure?

To look at the complete picture I reached for my favorite tool for understanding any system: a threat model. Threat models are a great tool, and I find that creating a threat model helps me to better understand a system and how to secure it. If you’ve never heard of threat models, check out these great descriptions.

Before I show you what I came up with, I want to share my biggest takeaway: you can’t make a threat model for all AI agents. It’s like making a threat model for all software – the details of the individual application matter.

What is an AI agent? I’m talking here about any autonomous or semi-autonomous program that interacts with large language models (LLMs) to perform some action. The program is able to operate somewhat independently, with the user and agent designer setting the goal and providing tools, but not providing a deterministic algorithm for the agent.

The problem with defining a threat model for “AI agents” is that the risks associated with running an agent is a product of the stuff the agent has access to, the stuff the agent is allowed to do, and who can provide input to the agent. And all of those factors vary a lot between agents. An agent that can both read and write emails from your account has a very different profile from an agent that can only read emails. 

The best case would be that everyone creates a good threat model for every agent that they create. That way the threat model could actually account for both what that agent can do, and also what resources the agent has access to. This would be great, and if you are deploying an agent in production I absolutely encourage you to do this yourself. Grab a whiteboard, a friend, and a tea or coffee and think about each component of your agent and what the risks are if they are compromised.

But that doesn’t help with my original question: what security properties do we need for AI agents?

To answer this question, it’s still worth considering a threat model for a generic AI agent. This is not a replacement for threat modeling the individual agents you create, but it can start to clear up systemic risks to these agents. This in turn can help us build the right security toolkit that can actually secure individual agents.

The STRIDE Threat Model for AI Agents

The diagram below shows my STRIDE-inspired threat model of AI agents. To utilize best practices, this model assumes that an agent uses some sort of interface to interact with external resources, and that the agent itself does not have access to credentials. For simplicity it does not consider sub-agents or multi-agent architectures and I considered a malicious or attacker-controlled model out of scope for this analysis.

There isn’t a single key takeaway from this process, but a few things to note:

• Tampering can happen at many stages: self-tampering within the agent (aka hallucination or loss of context), input tampering, and output tampering. This points to controls like those used for SQL injection prevention to ensure good inputs and outputs
• The reputation risk of AI agents, especially those with direct or indirect access to credentials, is a big open question. Who is responsible for the outputs of the agent?
• Which resources you give the agent matters a lot for information disclosure and tampering at each stage. This goes back to the discussion of the futility of this exercise. The less you give the agent, the less it can do if compromised or hallucinating.
• Privilege escalation is possible at several points: this is where the agent might attack itself just to be “helpful”, so needs to be hardened even if all inputs are trusted
• The non-deterministic nature of agents means that any controls must be hard boundaries, not politely asking the agent not to do something. We’ve seen several attacks that bypass “soft” controls in the agent’s instructions. Don’t rely on these. Use real, enforced, audited boundaries.

Mapping AI Agent Threats to Security Controls

Here is a very brief discussion of the threats identified above. Addressing all of these threats holistically is beyond the scope of this post, but I included my thoughts about which approaches could be useful in each case.

Where Edera Fits in AI Agent Security

I’ll talk more about this in a future post, but Edera provides a strong sandbox environment for AI agents including hardware enforced memory segmentation, built-in identity management, and a native object capability system. When combined with good system design, input and output filtering, and supply chain best practices for all tooling, Edera provides a strong base for AI agent security.